Trace number 3263586

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.27 1203.37

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_60_29_30_6-dir.shuffled-as.sat03-130.cnf
MD5SUMe826117d87949a5b285e72732bfa395b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables870
Number of clauses69964
Sum of the clauses size140740
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 269935
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 529

Solver Data 

0.02/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263586-1304097280.cnf 
0.02/0.04	c Using SATTIMEOUT of 4800
0.02/0.04	c Using SATRAM of 15500
0.02/0.04	c Size of subset node = 32
0.02/0.04	c Variables = 870
0.02/0.04	c Clauses = 69964
0.02/0.04	c Literals = 140740
0.02/0.04	c Reduced size:
0.02/0.04	c Variables = 870
0.02/0.04	c Clauses = 69964
0.02/0.04	c Literals = 140740
0.02/0.04	c a2
0.02/0.09	c a1
0.09/0.19	c Independent sub-problems = 1
0.09/0.19	c Sub-problem with 870 variables and 69964 clauses
0.09/0.19	c using permutation method 4
0.09/0.19	c count = 1
0.09/0.19	c Ordering finished 0.170 seconds
0.09/0.19	c mapped 0.180 seconds
0.09/0.19	c permuted 0.180 seconds
0.09/0.19	c about to sort
0.09/0.19	c sorted
0.29/0.36	c clauses now 69964
0.29/0.36	c init finished 0.200 seconds
0.29/0.36	c tree built 0.350 seconds
0.29/0.36	c 
0.29/0.36	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.36	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.36	c ................................ | ..........                    |                                  0.3         0.0             
0.29/0.36	c ................................ | ..........                  1 |                                  0.4         2.8             
0.29/0.36	c ................................ | ..........                  2 |                                  0.4         5.6             
0.29/0.36	c ................................ | ..........                  3 |                                  0.4         8.3             
0.29/0.36	c ................................ | ..........                  4 |                                  0.4        11.1             
0.29/0.36	c ................................ | ..........                  5 |                                  0.4        13.9             
0.29/0.36	c ................................ | ..........               1000 |              42                  0.4      2777.8         1036
95.87/24.53	c ................................ | ..........            6121000 | 107156       276517             96.0     63753.8     11097873
191.85/48.64	c ................................ | ..........           10080000 | 215196       481799            192.0     52497.3     19576112
287.75/72.76	c ................................ | ..........           14154000 | 321318       663017            288.0     49145.8     26039954
383.73/96.86	c ................................ | ..........           17515000 | 415753       824303            384.0     45612.0     32111310
479.78/120.96	c ................................ | *.........           20010000 | 507626       945492            480.0     41686.6     37833916
575.78/145.05	c ................................ | *.........           23860000 | 595558       1085849           576.0     41423.6     42626727
671.60/169.09	c ................................ | *.........           27309000 | 692618       1233320           672.0     40638.4     44430881
767.83/193.15	c ................................ | *.........           30575000 | 780368       1397835           768.0     39810.2     48559442
863.66/217.19	c ................................ | *.........           33667000 | 857061       1559957           864.0     38965.1     53684683
959.86/241.23	c ................................ | **........           36650000 | 946509       1697977           960.0     38176.3     57380509
1055.71/265.28	c ................................ | **........           39223000 | 1018293      1814169          1056.0     37141.9     60164098
1151.94/289.33	c ................................ | **........           41024000 | 1084592      1902349          1152.1     35609.3     62255427
1247.79/313.35	c ................................ | **........           42771000 | 1145164      2018436          1248.0     34271.4     66210234
1343.64/337.39	c ................................ | **........           44570000 | 1203081      2134168          1344.0     33161.7     70151287
1439.90/361.49	c ................................ | ***.......           46739000 | 1274660      2226004          1440.0     32457.2     73548373
1535.73/385.47	c ................................ | ***.......           48919000 | 1348799      2313817          1536.0     31848.1     74902945
1631.96/409.51	c ................................ | ***.......           50835000 | 1414029      2404651          1632.0     31148.7     79229929
1727.78/433.56	c ................................ | ***.......           53091000 | 1486917      2523243          1728.0     30723.8     83289874
1823.99/457.61	c ................................ | ***.......           54629000 | 1545197      2616825          1824.1     29949.0     83846281
1919.83/481.68	c ................................ | ****......           55748000 | 1600352      2703063          1920.2     29033.0     87491499
2015.63/505.70	c ................................ | ****......           56801000 | 1650254      2786275          2016.0     28174.7     90248873
2111.90/529.73	c ................................ | ****......           57833000 | 1696711      2854833          2112.0     27382.8     92240036
2207.68/553.78	c ................................ | ****......           59071000 | 1747210      2923605          2208.0     26752.8     94361749
2303.90/577.84	c ................................ | ****......           60041000 | 1793806      2985221          2304.1     26058.4     97029152
2399.75/601.87	c ................................ | *****.....           60963000 | 1838065      3037772          2400.1     25400.7     99688901
2496.00/625.90	c ................................ | *****.....           61833000 | 1884050      3096993          2496.0     24772.6    102264690
2591.82/649.96	c ................................ | *****.....           62828000 | 1939086      3158168          2592.1     24238.4    104589257
2688.03/674.01	c ................................ | *****.....           63668000 | 1982703      3216660          2688.1     23685.2    104714476
2783.86/698.04	c ................................ | *****.....           64393000 | 2029846      3268485          2784.0     23129.6    106576308
2879.67/722.09	c ................................ | ******....           64966000 | 2071574      3322935          2880.1     22557.2    108593445
2975.92/746.12	c ................................ | ******....           65785000 | 2113290      3370629          2976.0     22105.2    109661278
3072.16/770.20	c ................................ | ******....           66703000 | 2159161      3416878          3072.2     21711.9    112141364
3168.02/794.21	c ................................ | ******....           67500000 | 2198579      3473490          3168.1     21306.3    115268855
3263.87/818.25	c ................................ | ******....           68158000 | 2233820      3540053          3264.1     20881.1    119592647
3359.65/842.29	c ................................ | *******...           69509000 | 2292298      3631573          3360.0     20687.1    123823966
3455.87/866.34	c ................................ | *******...           70664000 | 2342678      3708660          3456.0     20446.5    127365457
3551.68/890.39	c ................................ | *******...           71400000 | 2382545      3775273          3552.1     20101.1    130557125
3647.92/914.44	c ................................ | *******...           72296000 | 2423381      3851557          3648.1     19817.5    133478821
3743.77/938.45	c ................................ | *******...           73149000 | 2461426      3915384          3744.0     19537.6    134050759
3839.99/962.50	c ................................ | ********..           73891000 | 2496853      3991113          3840.0     19242.4    137561000
3935.81/986.57	c ................................ | ********..           74844000 | 2537560      4094989          3936.1     19014.7    140371404
4032.01/1010.67	c ................................ | ********..           75458000 | 2575193      4180615          4032.2     18713.9    142215032
4128.06/1034.71	c ................................ | ********..           76238000 | 2615024      4260758          4128.1     18468.1    143771239
4224.17/1058.81	c ................................ | ********..           77273000 | 2656546      4333318          4224.1     18293.3    145134175
4320.29/1082.91	c ................................ | *********.           78195000 | 2700013      4391601          4320.1     18100.5    147099599
4416.07/1106.99	c ................................ | *********.           79082000 | 2743130      4468461          4416.1     17907.8    150642616
4511.80/1130.99	c ................................ | *********.           80029000 | 2784329      4541196          4512.1     17736.6    153056967
4607.95/1155.09	c ................................ | *********.           80962000 | 2828960      4610164          4608.1     17569.5    156174675
4704.04/1179.12	c ................................ | *********.           82288000 | 2875694      4688498          4704.0     17493.2    157594688
4799.79/1203.16	c CPU limit (4800) reached
4799.79/1203.16	c Time = 4800.05 seconds
4799.79/1203.16	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263586-1304097280/watcher-3263586-1304097280 -o /tmp/evaluation-result-3263586-1304097280/solver-3263586-1304097280 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263586-1304097280.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.11 7.64 7.69 2/175 12966
/proc/meminfo: memFree=26621012/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) R 12964 12966 12928 0 -1 4202496 465 0 0 0 0 0 0 0 25 0 1 0 521326658 9768960 408 33554432000 4194304 4338100 140733541637808 18446744073709551615 242970211013 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/12966/statm: 2385 409 147 36 0 290 0

[startup+0.0305031 s]
/proc/loadavg: 6.11 7.64 7.69 2/175 12966
/proc/meminfo: memFree=26621012/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=14464 CPUtime=0.02 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) R 12964 12966 12928 0 -1 4202496 1674 0 0 0 2 0 0 0 25 0 1 0 521326658 14811136 1617 33554432000 4194304 4338100 140733541637808 18446744073709551615 242970211013 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/12966/statm: 3616 1620 148 36 0 1521 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 14464

[startup+0.100524 s]
/proc/loadavg: 6.11 7.64 7.69 2/175 12966
/proc/meminfo: memFree=26621012/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=23904 CPUtime=0.09 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) R 12964 12966 12928 0 -1 4202496 3935 0 0 0 9 0 0 0 25 0 1 0 521326658 24477696 3877 33554432000 4194304 4338100 140733541637808 18446744073709551615 4228108 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/12966/statm: 5976 3878 153 36 0 3881 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 23904

[startup+0.300429 s]
/proc/loadavg: 6.11 7.64 7.69 2/175 12966
/proc/meminfo: memFree=26621012/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=43996 CPUtime=0.29 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) R 12964 12966 12928 0 -1 4202496 10523 0 0 0 28 1 0 0 25 0 1 0 521326658 45051904 9027 33554432000 4194304 4338100 140733541637808 18446744073709551615 242970211013 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/12966/statm: 10999 9027 164 36 0 8904 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 43996

[startup+1.17624 s]
/proc/loadavg: 6.11 7.64 7.69 2/175 12966
/proc/meminfo: memFree=26621012/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=253976 CPUtime=3.59 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 17720 0 0 0 356 3 0 0 25 0 5 0 521326658 260071424 16224 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 63494 16224 176 36 0 61399 0
Current children cumulated CPU time (s) 3.59
Current children cumulated vsize (KiB) 253976

[startup+1.50018 s]
/proc/loadavg: 6.11 7.64 7.69 2/175 12966
/proc/meminfo: memFree=26621012/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=253976 CPUtime=4.88 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 19250 0 0 0 485 3 0 0 25 0 5 0 521326658 260071424 17754 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 63494 17755 176 36 0 61399 0
Current children cumulated CPU time (s) 4.88
Current children cumulated vsize (KiB) 253976

[startup+3.1009 s]
/proc/loadavg: 6.11 7.64 7.69 10/184 12975
/proc/meminfo: memFree=26422940/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=282524 CPUtime=11.26 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 26453 0 0 0 1120 6 0 0 25 0 5 0 521326658 289304576 24957 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 70631 24957 176 36 0 68536 0
[pid=12966/tid=12972] ppid=12964 vsize=282524 CPUtime=2.73 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 1689 0 0 0 271 2 0 0 24 0 5 0 521326695 289304576 24957 33554432000 4194304 4338100 140733541637808 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=282524 CPUtime=2.73 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 4646 0 0 0 272 1 0 0 23 0 5 0 521326695 289304576 24957 33554432000 4194304 4338100 140733541637808 18446744073709551615 242969945842 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=282524 CPUtime=2.7 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 2715 0 0 0 269 1 0 0 18 0 5 0 521326695 289304576 24957 33554432000 4194304 4338100 140733541637808 18446744073709551615 4286715 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=282524 CPUtime=2.72 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 4024 0 0 0 272 0 0 0 25 0 5 0 521326695 289304576 24957 33554432000 4194304 4338100 140733541637808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.26
Current children cumulated vsize (KiB) 282524

[startup+6.30036 s]
/proc/loadavg: 6.34 7.67 7.70 10/184 12975
/proc/meminfo: memFree=26305536/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=329952 CPUtime=23.41 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 37392 0 0 0 2323 18 0 0 25 0 5 0 521326658 337870848 35896 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 82488 35897 176 36 0 80393 0
[pid=12966/tid=12972] ppid=12964 vsize=329952 CPUtime=5.88 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 3660 0 0 0 583 5 0 0 18 0 5 0 521326695 337870848 35897 33554432000 4194304 4338100 140733541637808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=329952 CPUtime=5.45 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 7927 0 0 0 543 2 0 0 25 0 5 0 521326695 337870848 35897 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=329952 CPUtime=5.76 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 4980 0 0 0 570 6 0 0 22 0 5 0 521326695 337870848 35897 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=329952 CPUtime=5.92 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 7447 0 0 0 590 2 0 0 25 0 5 0 521326695 337870848 35897 33554432000 4194304 4338100 140733541637808 18446744073709551615 4240519 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 23.41
Current children cumulated vsize (KiB) 329952

[startup+12.7063 s]
/proc/loadavg: 6.47 7.67 7.70 9/184 12975
/proc/meminfo: memFree=26097072/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=430960 CPUtime=48.91 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 62572 0 0 0 4856 35 0 0 25 0 5 0 521326658 441303040 61076 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 107740 61076 176 36 0 105645 0
[pid=12966/tid=12972] ppid=12964 vsize=430960 CPUtime=12.21 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 7599 0 0 0 1211 10 0 0 18 0 5 0 521326695 441303040 61076 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=430960 CPUtime=11.82 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 15202 0 0 0 1179 3 0 0 25 0 5 0 521326695 441303040 61076 33554432000 4194304 4338100 140733541637808 18446744073709551615 242969945464 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=430960 CPUtime=12.17 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 9593 0 0 0 1202 15 0 0 25 0 5 0 521326695 441303040 61076 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=430960 CPUtime=12.33 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 16799 0 0 0 1229 4 0 0 25 0 5 0 521326695 441303040 61076 33554432000 4194304 4338100 140733541637808 18446744073709551615 4285738 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 48.91
Current children cumulated vsize (KiB) 430960

[startup+25.501 s]
/proc/loadavg: 6.85 7.72 7.71 9/184 12976
/proc/meminfo: memFree=25753496/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=594748 CPUtime=99.86 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 104694 0 0 0 9918 68 0 0 25 0 5 0 521326658 609021952 103198 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 148687 103198 176 36 0 146592 0
[pid=12966/tid=12972] ppid=12964 vsize=594748 CPUtime=24.82 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 14731 0 0 0 2461 21 0 0 18 0 5 0 521326695 609021952 103199 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=594748 CPUtime=24.56 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 26186 0 0 0 2450 6 0 0 25 0 5 0 521326695 609021952 103199 33554432000 4194304 4338100 140733541637808 18446744073709551615 242969945153 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=594748 CPUtime=24.96 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 18218 0 0 0 2464 32 0 0 25 0 5 0 521326695 609021952 103199 33554432000 4194304 4338100 140733541637808 18446744073709551615 4285849 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=594748 CPUtime=25.12 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 32181 0 0 0 2506 6 0 0 25 0 5 0 521326695 609021952 103199 33554432000 4194304 4338100 140733541637808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 99.86
Current children cumulated vsize (KiB) 594748

[startup+51.1006 s]
/proc/loadavg: 7.41 7.77 7.73 9/184 12976
/proc/meminfo: memFree=25222008/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=883496 CPUtime=201.81 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 176344 0 0 0 20049 132 0 0 25 0 5 0 521326658 904699904 174848 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 220874 174848 176 36 0 218779 0
[pid=12966/tid=12972] ppid=12964 vsize=883496 CPUtime=50.11 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 27762 0 0 0 4970 41 0 0 18 0 5 0 521326695 904699904 174850 33554432000 4194304 4338100 140733541637808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=883496 CPUtime=50.03 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 51344 0 0 0 4990 13 0 0 25 0 5 0 521326695 904699904 174851 33554432000 4194304 4338100 140733541637808 18446744073709551615 242969945337 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=883496 CPUtime=50.55 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 31979 0 0 0 4990 65 0 0 25 0 5 0 521326695 904699904 174851 33554432000 4194304 4338100 140733541637808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=883496 CPUtime=50.72 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 51883 0 0 0 5062 10 0 0 25 0 5 0 521326695 904699904 174851 33554432000 4194304 4338100 140733541637808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 201.81
Current children cumulated vsize (KiB) 883496

[startup+102.301 s]
/proc/loadavg: 7.79 7.82 7.75 9/184 12978
/proc/meminfo: memFree=24452204/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=1293856 CPUtime=405.67 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 280038 0 0 0 40330 237 0 0 25 0 5 0 521326658 1324908544 278542 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 323464 278542 176 36 0 321369 0
[pid=12966/tid=12972] ppid=12964 vsize=1293856 CPUtime=100.84 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 51651 0 0 0 10001 83 0 0 25 0 5 0 521326695 1324908544 278542 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=1293856 CPUtime=100.95 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 76151 0 0 0 10076 19 0 0 25 0 5 0 521326695 1324908544 278542 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=1293856 CPUtime=101.57 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 58592 0 0 0 10041 116 0 0 18 0 5 0 521326695 1324908544 278542 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=1293856 CPUtime=101.89 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 80265 0 0 0 10174 15 0 0 25 0 5 0 521326695 1324908544 278542 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 405.67
Current children cumulated vsize (KiB) 1293856

[startup+162.306 s]
/proc/loadavg: 7.97 7.86 7.76 9/184 12979
/proc/meminfo: memFree=23743904/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=1661216 CPUtime=644.89 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 368755 0 0 0 64030 459 0 0 25 0 5 0 521326658 1701085184 367259 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 415304 367259 176 36 0 413209 0
[pid=12966/tid=12972] ppid=12964 vsize=1661216 CPUtime=160.72 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 73332 0 0 0 15861 211 0 0 25 0 5 0 521326695 1701085184 367259 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=1661216 CPUtime=160.75 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 99805 0 0 0 16049 26 0 0 25 0 5 0 521326695 1701085184 367259 33554432000 4194304 4338100 140733541637808 18446744073709551615 242969945513 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=1661216 CPUtime=161.18 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 83055 0 0 0 15922 196 0 0 25 0 5 0 521326695 1701085184 367259 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=1661216 CPUtime=161.83 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 99184 0 0 0 16161 22 0 0 25 0 5 0 521326695 1701085184 367259 33554432000 4194304 4338100 140733541637808 18446744073709551615 242969945140 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 644.89
Current children cumulated vsize (KiB) 1661216

[startup+222.301 s]
/proc/loadavg: 7.99 7.89 7.78 9/184 12981
/proc/meminfo: memFree=23184276/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=1988508 CPUtime=884.41 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 452385 0 0 0 87581 860 0 0 25 0 5 0 521326658 2036232192 450889 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 497127 450889 176 36 0 495032 0
[pid=12966/tid=12972] ppid=12964 vsize=1988508 CPUtime=220.54 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 92402 0 0 0 21647 407 0 0 25 0 5 0 521326695 2036232192 450889 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 2 0 0 0

################
# More data... #
################

/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 237645 0 0 0 70095 63 0 0 25 0 5 0 521326695 3749306368 871392 33554432000 4194304 4338100 140733541637808 18446744073709551615 4279533 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2801.03
Current children cumulated vsize (KiB) 3661432

[startup+762.301 s]
/proc/loadavg: 7.79 7.98 7.87 9/184 12995
/proc/meminfo: memFree=20018748/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=3787212 CPUtime=3040.62 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 904873 0 0 0 301156 2906 0 0 25 0 5 0 521326658 3878105088 903377 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 946803 903377 176 36 0 944708 0
[pid=12966/tid=12972] ppid=12964 vsize=3787212 CPUtime=759.63 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 162814 0 0 0 74562 1401 0 0 25 0 5 0 521326695 3878105088 903377 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982362830 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=3787212 CPUtime=759.35 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 299139 0 0 0 75850 85 0 0 25 0 5 0 521326695 3878105088 903377 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=3787212 CPUtime=759.68 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 176706 0 0 0 74619 1349 0 0 25 0 5 0 521326695 3878105088 903377 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982364419 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=3787212 CPUtime=761.56 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 252835 0 0 0 76088 68 0 0 25 0 5 0 521326695 3878105088 903377 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982362830 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3040.62
Current children cumulated vsize (KiB) 3787212

[startup+822.307 s]
/proc/loadavg: 7.92 7.98 7.88 9/184 12997
/proc/meminfo: memFree=19677236/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=4072784 CPUtime=3280.25 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 977261 0 0 0 324967 3058 0 0 25 0 5 0 521326658 4170530816 975765 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 1018196 975765 176 36 0 1016101 0
[pid=12966/tid=12972] ppid=12964 vsize=4072784 CPUtime=819.63 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 172996 0 0 0 80495 1468 0 0 25 0 5 0 521326695 4170530816 975765 33554432000 4194304 4338100 140733541637808 18446744073709551615 4279533 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=4072784 CPUtime=819.32 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 328049 0 0 0 81838 94 0 0 25 0 5 0 521326695 4170530816 975765 33554432000 4194304 4338100 140733541637808 18446744073709551615 4240550 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=4072784 CPUtime=819.36 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 184385 0 0 0 80519 1417 0 0 25 0 5 0 521326695 4170530816 975766 33554432000 4194304 4338100 140733541637808 18446744073709551615 242969945838 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=4072784 CPUtime=821.56 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 278452 0 0 0 82079 77 0 0 25 0 5 0 521326695 4170530816 975766 33554432000 4194304 4338100 140733541637808 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3280.25
Current children cumulated vsize (KiB) 4072784

[startup+882.3 s]
/proc/loadavg: 8.02 7.99 7.89 9/184 12999
/proc/meminfo: memFree=19310924/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=4355756 CPUtime=3519.75 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 1048403 0 0 0 348700 3275 0 0 25 0 5 0 521326658 4460294144 1046907 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 1088939 1046908 176 36 0 1086844 0
[pid=12966/tid=12972] ppid=12964 vsize=4355756 CPUtime=879.57 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 187679 0 0 0 86385 1572 0 0 25 0 5 0 521326695 4460294144 1046908 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=4355756 CPUtime=879.26 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 353276 0 0 0 87824 102 0 0 25 0 5 0 521326695 4460294144 1046908 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=4355756 CPUtime=878.99 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 198106 0 0 0 86383 1516 0 0 25 0 5 0 521326695 4460294144 1046908 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=4355756 CPUtime=881.53 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 295964 0 0 0 88071 82 0 0 25 0 5 0 521326695 4460294144 1046908 33554432000 4194304 4338100 140733541637808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3519.75
Current children cumulated vsize (KiB) 4355756

[startup+942.301 s]
/proc/loadavg: 8.00 7.99 7.90 9/184 13000
/proc/meminfo: memFree=19022368/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=4519336 CPUtime=3759.35 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 1089362 0 0 0 372512 3423 0 0 25 0 5 0 521326658 4627800064 1087866 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 1129834 1087866 176 36 0 1127739 0
[pid=12966/tid=12972] ppid=12964 vsize=4519336 CPUtime=939.46 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 194888 0 0 0 92304 1642 0 0 25 0 5 0 521326695 4627800064 1087866 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=4519336 CPUtime=939.23 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 364084 0 0 0 93816 107 0 0 25 0 5 0 521326695 4627800064 1087866 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=4519336 CPUtime=938.74 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 203920 0 0 0 92290 1584 0 0 25 0 5 0 521326695 4627800064 1087866 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=4519336 CPUtime=941.53 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 313091 0 0 0 94066 87 0 0 25 0 5 0 521326695 4627800064 1087866 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3759.35
Current children cumulated vsize (KiB) 4519336

[startup+1002.3 s]
/proc/loadavg: 8.00 7.99 7.90 9/184 13002
/proc/meminfo: memFree=18685196/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=4752128 CPUtime=3998.89 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 1148549 0 0 0 396316 3573 0 0 25 0 5 0 521326658 4866179072 1147053 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 1188032 1147053 176 36 0 1185937 0
[pid=12966/tid=12972] ppid=12964 vsize=4752128 CPUtime=999.14 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 201664 0 0 0 98200 1714 0 0 25 0 5 0 521326695 4866179072 1147053 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=4752128 CPUtime=999.18 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 379618 0 0 0 99806 112 0 0 25 0 5 0 521326695 4866179072 1147053 33554432000 4194304 4338100 140733541637808 18446744073709551615 4284181 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=4752128 CPUtime=998.69 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 209502 0 0 0 98220 1649 0 0 18 0 5 0 521326695 4866179072 1147053 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=4752128 CPUtime=1001.49 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 344386 0 0 0 100054 95 0 0 25 0 5 0 521326695 4866179072 1147053 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3998.89
Current children cumulated vsize (KiB) 4752128

[startup+1062.3 s]
/proc/loadavg: 8.00 7.99 7.91 9/184 13003
/proc/meminfo: memFree=18529940/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=4862532 CPUtime=4238.09 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 1176450 0 0 0 420184 3625 0 0 25 0 5 0 521326658 4979232768 1174954 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 1215633 1174954 176 36 0 1213538 0
[pid=12966/tid=12972] ppid=12964 vsize=4862532 CPUtime=1059.06 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 206183 0 0 0 104153 1753 0 0 25 0 5 0 521326695 4979232768 1174954 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=4862532 CPUtime=1059.13 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 392913 0 0 0 105795 118 0 0 25 0 5 0 521326695 4979232768 1174954 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=4862532 CPUtime=1058.4 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 209765 0 0 0 104186 1654 0 0 18 0 5 0 521326695 4979232768 1174954 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=4862532 CPUtime=1061.11 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 354210 0 0 0 106013 98 0 0 25 0 5 0 521326695 4979232768 1174954 33554432000 4194304 4338100 140733541637808 18446744073709551615 4255718 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4238.09
Current children cumulated vsize (KiB) 4862532

[startup+1122.3 s]
/proc/loadavg: 8.07 8.02 7.92 9/184 13005
/proc/meminfo: memFree=18308836/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=5076292 CPUtime=4477.52 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 1229629 0 0 0 444067 3685 0 0 25 0 5 0 521326658 5198123008 1228133 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 1269073 1228133 176 36 0 1266978 0
[pid=12966/tid=12972] ppid=12964 vsize=5076292 CPUtime=1119.06 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 218422 0 0 0 110112 1794 0 0 25 0 5 0 521326695 5198123008 1228133 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=5076292 CPUtime=1119.09 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 403117 0 0 0 111785 124 0 0 25 0 5 0 521326695 5198123008 1228133 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=5076292 CPUtime=1118.17 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 221388 0 0 0 110156 1661 0 0 18 0 5 0 521326695 5198123008 1228133 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=5076292 CPUtime=1120.81 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 373323 0 0 0 111978 103 0 0 25 0 5 0 521326695 5198123008 1228133 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4477.52
Current children cumulated vsize (KiB) 5076292

[startup+1182.3 s]
/proc/loadavg: 8.17 8.06 7.94 9/184 13007
/proc/meminfo: memFree=18007624/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=5258640 CPUtime=4716.8 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) S 12964 12966 12928 0 -1 4202496 1275745 0 0 0 467939 3741 0 0 25 0 5 0 521326658 5384847360 1274249 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 1314660 1274249 176 36 0 1312565 0
[pid=12966/tid=12972] ppid=12964 vsize=5258640 CPUtime=1179.03 cores=0,2,4,6
/proc/12966/task/12972/stat : 12972 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 227449 0 0 0 116069 1834 0 0 25 0 5 0 521326695 5384847360 1274249 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12966/tid=12973] ppid=12964 vsize=5258640 CPUtime=1178.98 cores=0,2,4,6
/proc/12966/task/12973/stat : 12973 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 414793 0 0 0 117769 129 0 0 25 0 5 0 521326695 5384847360 1274249 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=5258640 CPUtime=1177.87 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 229955 0 0 0 116119 1668 0 0 18 0 5 0 521326695 5384847360 1274249 33554432000 4194304 4338100 140733541637808 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12966/tid=12975] ppid=12964 vsize=5258640 CPUtime=1180.53 cores=0,2,4,6
/proc/12966/task/12975/stat : 12975 (ttsth-5-4) R 12964 12966 12928 0 -1 4202560 390169 0 0 0 117946 107 0 0 25 0 5 0 521326695 5384847360 1274249 33554432000 4194304 4338100 140733541637808 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4716.8
Current children cumulated vsize (KiB) 5258640



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.2 s]
/proc/loadavg: 8.12 8.06 7.94 2/175 13008
/proc/meminfo: memFree=22267556/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=0 CPUtime=4800.1 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) Z 12964 12966 12928 0 -1 4203548 1281438 0 0 0 476197 3813 0 0 18 0 2 0 521326658 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 0 0 0 0 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=0 CPUtime=1198.7 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4203588 232278 0 0 0 118169 1701 0 0 25 0 2 0 521326695 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.31 s]
/proc/loadavg: 8.12 8.06 7.94 2/175 13008
/proc/meminfo: memFree=22267556/32951132 swapFree=67111364/67111528
[pid=12966] ppid=12964 vsize=0 CPUtime=4800.21 cores=0,2,4,6
/proc/12966/stat : 12966 (ttsth-5-4) Z 12964 12966 12928 0 -1 4203548 1281438 0 0 0 476197 3824 0 0 18 0 2 0 521326658 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12966/statm: 0 0 0 0 0 0 0
[pid=12966/tid=12974] ppid=12964 vsize=0 CPUtime=1198.8 cores=0,2,4,6
/proc/12966/task/12974/stat : 12974 (ttsth-5-4) R 12964 12966 12928 0 -1 4203588 232278 0 0 0 118169 1711 0 0 25 0 2 0 521326695 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.37
CPU time (s): 4800.27
CPU user time (s): 4761.97
CPU system time (s): 38.3012
CPU usage (%): 398.901
Max. virtual memory (cumulated for all children) (KiB): 5288044

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4761.97
system time used= 38.3012
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1281438
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 10759
involuntary context switches= 20508

runsolver used 1.93771 second user time and 4.22836 second system time

The end

Launcher Data 

Begin job on node128 at 2011-04-29 19:14:40
IDJOB=3263586
IDBENCH=83986
IDSOLVER=1654
FILE ID=node128/3263586-1304097280
RUNJOBID= node128-1304097280-12946
PBS_JOBID= 13170611
Free space on /tmp= 66972 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_60_29_30_6-dir.shuffled-as.sat03-130.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263586-1304097280/watcher-3263586-1304097280 -o /tmp/evaluation-result-3263586-1304097280/solver-3263586-1304097280 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263586-1304097280.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e826117d87949a5b285e72732bfa395b
RANDOM SEED=1156877426

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      26621788 kB
Buffers:       1281608 kB
Cached:        4392172 kB
SwapCached:        160 kB
Active:         339168 kB
Inactive:      5401292 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      26621788 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:            1340 kB
Writeback:           8 kB
AnonPages:       66068 kB
Mapped:          14692 kB
Slab:           523960 kB
PageTables:       4692 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   209572 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66972 MiB
End job on node128 at 2011-04-29 19:34:45