Trace number 3263718

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.29 1204.83

General information on the benchmark

NameSAT03/handmade/anton/SAT/
SGI_30_80_15_90_4-dir.shuffled-as.sat03-6.cnf
MD5SUM22bbcb4a25a1da66113c240898c75509
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.19197
Satisfiable
(Un)Satisfiability was proved
Number of variables450
Number of clauses28508
Sum of the clauses size57436
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 228493
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 515

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263718-1304097473.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 450
0.00/0.02	c Clauses = 28508
0.00/0.02	c Literals = 57436
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 450
0.00/0.02	c Clauses = 28508
0.00/0.02	c Literals = 57436
0.00/0.02	c a2
0.00/0.04	c a1
0.06/0.07	c Independent sub-problems = 1
0.06/0.07	c Sub-problem with 450 variables and 28508 clauses
0.06/0.07	c using permutation method 4
0.06/0.07	c count = 1
0.06/0.07	c Ordering finished 0.060 seconds
0.06/0.07	c mapped 0.060 seconds
0.06/0.07	c permuted 0.060 seconds
0.06/0.07	c about to sort
0.06/0.07	c sorted
0.09/0.13	c clauses now 28508
0.09/0.13	c init finished 0.070 seconds
0.09/0.13	c tree built 0.120 seconds
0.09/0.13	c 
0.09/0.13	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.13	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.13	c ................................ | ..........                    |                                  0.1         0.0             
0.09/0.13	c ................................ | ..........                  1 |                                  0.1         7.7             
0.09/0.13	c ................................ | ..........                  2 |                                  0.1        15.4             
0.09/0.13	c ................................ | ..........                  3 |                                  0.1        23.1             
0.09/0.13	c ................................ | ..........                  4 |                                  0.1        30.8             
0.09/0.13	c ................................ | ..........                  5 |                                  0.1        38.5             
0.09/0.13	c ................................ | ..........               1000 | 1            86                  0.1      7692.3         4136
95.98/24.21	c ................................ | ..........            5709000 | 84318        583675             96.0     59462.6     12748701
191.93/48.32	c ................................ | ..........            9590000 | 155080       1037679           192.0     49945.3     22558719
287.97/72.41	c ................................ | ..........           12611000 | 231801       1328697           288.0     43786.7     26886677
383.92/96.52	c ................................ | ..........           16603000 | 300599       1777584           384.0     43235.9     32200395
479.87/120.63	c ................................ | *.........           20800000 | 433078       2089094           480.0     43331.5     39652092
575.93/144.72	c ................................ | *.........           24077000 | 552355       2274155           576.0     41798.2     44947298
671.93/168.82	c ................................ | *.........           26899000 | 664453       2502658           672.0     40027.7     49939684
767.91/192.92	c ................................ | *.........           29431000 | 795433       2711698           768.0     38321.6     57066507
863.92/217.02	c ................................ | *.........           31472000 | 906307       2924760           864.0     36425.5     62839280
959.94/241.13	c ................................ | **........           32956000 | 1037554      3179444           960.1     34327.0     69211816
1055.93/265.22	c ................................ | **........           34650000 | 1174908      3331585          1056.0     32811.9     72322435
1151.96/289.32	c ................................ | **........           36582000 | 1287218      3452522          1152.0     31754.4     74798705
1247.90/313.44	c ................................ | **........           38556000 | 1383410      3597431          1248.0     30893.5     77683301
1343.83/337.56	c ................................ | **........           40420000 | 1484063      3761296          1344.1     30072.2     78718297
1439.88/361.63	c ................................ | ***.......           42603000 | 1575712      3913905          1440.0     29585.4     82819797
1535.87/385.74	c ................................ | ***.......           44532000 | 1652050      4014281          1536.0     28991.6     84637275
1631.83/409.84	c ................................ | ***.......           46451000 | 1729483      4111424          1632.0     28462.6     86642184
1727.86/433.93	c ................................ | ***.......           48485000 | 1807659      4254453          1728.0     28058.4     89408559
1823.92/458.02	c ................................ | ***.......           50466000 | 1846896      4403619          1824.0     27667.8     91010467
1919.91/482.13	c ................................ | ****......           53809000 | 1995831      4527545          1920.0     28025.1     96170486
2015.94/506.21	c ................................ | ****......           56284000 | 2199494      4596367          2016.0     27918.7     98375918
2111.93/530.31	c ................................ | ****......           59232000 | 2372268      4688067          2112.0     28045.3    101604488
2207.95/554.42	c ................................ | ****......           61898000 | 2534424      4777168          2208.0     28033.4    104537572
2303.70/578.48	c ................................ | ****......           63581000 | 2692789      4862843          2304.0     27595.8    107402415
2399.76/602.56	c ................................ | *****.....           66031000 | 2858324      4962999          2400.0     27512.8    113439204
2495.79/626.67	c ................................ | *****.....           67900000 | 3016284      5047744          2496.1     27202.9    115755975
2591.93/650.72	c ................................ | *****.....           69876000 | 3166334      5099674          2592.0     26958.2    116604948
2687.95/674.81	c ................................ | *****.....           72501000 | 3330728      5188338          2688.0     26972.0    120035892
2784.01/698.90	c ................................ | *****.....           74675000 | 3484700      5260907          2784.0     26822.8    123099478
2879.65/722.99	c ................................ | ******....           77158000 | 3631780      5342722          2880.0     26790.8    126009055
2975.81/747.04	c ................................ | ******....           78996000 | 3781165      5452435          2976.0     26544.4    128212940
3071.87/771.14	c ................................ | ******....           81044000 | 3898244      5517813          3072.0     26381.3    131356842
3167.97/795.22	c ................................ | ******....           83039000 | 4032910      5593487          3168.1     26211.4    135283939
3263.64/819.29	c ................................ | ******....           84406000 | 4168854      5650489          3264.0     25859.7    135937412
3359.99/843.42	c ................................ | *******...           86541000 | 4303490      5727158          3360.1     25755.6    140138203
3455.68/867.48	c ................................ | *******...           88360000 | 4437671      5810659          3456.0     25567.1    141290264
3551.80/891.55	c ................................ | *******...           90208000 | 4571960      5900219          3552.0     25396.3    143842358
3647.98/915.60	c ................................ | *******...           91965000 | 4660830      6011596          3648.0     25209.7    146984983
3743.73/939.67	c ................................ | *******...           93515000 | 4778599      6089936          3744.0     24977.1    150870763
3839.76/963.76	c ................................ | ********..           94983000 | 4875486      6172005          3840.0     24735.0    153754453
3935.86/987.84	c ................................ | ********..           96305000 | 4994173      6237679          3936.0     24467.6    154682559
4032.04/1011.92	c ................................ | ********..           97236000 | 5094352      6301475          4032.1     24115.5    155784787
4127.72/1035.97	c ................................ | ********..           98413000 | 5213172      6357433          4128.0     23840.1    156679722
4223.88/1060.05	c ................................ | ********..           99363000 | 5321213      6414258          4224.1     23522.9    157798661
4320.00/1084.12	c ................................ | *********.          100725000 | 5452566      6464864          4320.1     23315.4    159876686
4415.96/1108.21	c ................................ | *********.          102118000 | 5578147      6508285          4416.0     23124.5    161090010
4511.93/1132.32	c ................................ | *********.          103488000 | 5701015      6546991          4512.0     22936.0    161090010
4607.97/1156.41	c ................................ | *********.          104945000 | 5816403      6590820          4608.0     22774.4    162134039
4704.00/1180.50	c ................................ | *********.          106509000 | 5928320      6640950          4704.0     22642.1    162757897
4799.69/1204.59	c CPU limit (4800) reached
4799.69/1204.59	c Time = 4800.05 seconds
4799.69/1204.59	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263718-1304097473/watcher-3263718-1304097473 -o /tmp/evaluation-result-3263718-1304097473/solver-3263718-1304097473 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263718-1304097473.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.06 8.03 7.63 2/177 21703
/proc/meminfo: memFree=30862732/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) R 21701 21703 21560 0 -1 4202496 413 0 0 0 0 0 0 0 25 0 1 0 599977893 9633792 355 33554432000 4194304 4338100 140734566556384 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21703/statm: 2352 356 147 36 0 257 0

[startup+0.067606 s]
/proc/loadavg: 8.06 8.03 7.63 2/177 21703
/proc/meminfo: memFree=30862732/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=15100 CPUtime=0.06 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) R 21701 21703 21560 0 -1 4202496 2255 0 0 0 6 0 0 0 25 0 1 0 599977893 15462400 1812 33554432000 4194304 4338100 140734566556384 18446744073709551615 4295674 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21703/statm: 3775 1812 156 36 0 1680 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 15100

[startup+0.100605 s]
/proc/loadavg: 8.06 8.03 7.63 2/177 21703
/proc/meminfo: memFree=30862732/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=21852 CPUtime=0.09 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) R 21701 21703 21560 0 -1 4202496 3963 0 0 0 9 0 0 0 25 0 1 0 599977893 22376448 3464 33554432000 4194304 4338100 140734566556384 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/21703/statm: 5463 3470 164 36 0 3368 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 21852

[startup+0.300532 s]
/proc/loadavg: 8.06 8.03 7.63 2/177 21703
/proc/meminfo: memFree=30862732/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=148264 CPUtime=0.77 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 7668 0 0 0 76 1 0 0 25 0 5 0 599977893 151822336 7169 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 37066 7169 176 36 0 34971 0
Current children cumulated CPU time (s) 0.77
Current children cumulated vsize (KiB) 148264

[startup+0.700463 s]
/proc/loadavg: 8.06 8.03 7.63 2/177 21703
/proc/meminfo: memFree=30862732/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=233128 CPUtime=2.37 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 11987 0 0 0 235 2 0 0 25 0 5 0 599977893 238723072 11488 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 58282 11488 176 36 0 56187 0
Current children cumulated CPU time (s) 2.37
Current children cumulated vsize (KiB) 233128

[startup+1.50031 s]
/proc/loadavg: 8.06 8.03 7.63 6/182 21708
/proc/meminfo: memFree=30804828/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=261676 CPUtime=5.56 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 18982 0 0 0 552 4 0 0 25 0 5 0 599977893 267956224 18483 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 65419 18484 176 36 0 63324 0
[pid=21703/tid=21705] ppid=21701 vsize=261676 CPUtime=1.35 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 3567 0 0 0 135 0 0 0 18 0 5 0 599977907 267956224 18484 33554432000 4194304 4338100 140734566556384 18446744073709551615 4240545 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=261676 CPUtime=1.35 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 750 0 0 0 135 0 0 0 20 0 5 0 599977907 267956224 18484 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=261676 CPUtime=1.35 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 4882 0 0 0 135 0 0 0 18 0 5 0 599977907 267956224 18484 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=261676 CPUtime=1.33 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 4088 0 0 0 133 0 0 0 18 0 5 0 599977907 267956224 18485 33554432000 4194304 4338100 140734566556384 18446744073709551615 260019791598 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 5.56
Current children cumulated vsize (KiB) 261676

[startup+3.10099 s]
/proc/loadavg: 7.82 7.98 7.62 10/186 21712
/proc/meminfo: memFree=30747204/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=300004 CPUtime=11.93 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 29465 0 0 0 1184 9 0 0 25 0 5 0 599977893 307204096 28966 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 75001 28968 176 36 0 72906 0
[pid=21703/tid=21705] ppid=21701 vsize=300004 CPUtime=2.95 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 6384 0 0 0 294 1 0 0 20 0 5 0 599977907 307204096 28968 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=300004 CPUtime=2.95 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 2120 0 0 0 293 2 0 0 25 0 5 0 599977907 307204096 28968 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=300004 CPUtime=2.96 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 8793 0 0 0 294 2 0 0 21 0 5 0 599977907 307204096 28968 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=300004 CPUtime=2.92 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 6474 0 0 0 290 2 0 0 18 0 5 0 599977907 307204096 28968 33554432000 4194304 4338100 140734566556384 18446744073709551615 4241581 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.93
Current children cumulated vsize (KiB) 300004

[startup+6.3004 s]
/proc/loadavg: 7.82 7.98 7.62 9/186 21712
/proc/meminfo: memFree=30486788/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=385812 CPUtime=24.68 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 50488 0 0 0 2451 17 0 0 25 0 5 0 599977893 395071488 49989 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 96453 49989 176 36 0 94358 0
[pid=21703/tid=21705] ppid=21701 vsize=385812 CPUtime=6.14 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 13400 0 0 0 612 2 0 0 25 0 5 0 599977907 395071488 49989 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=385812 CPUtime=6.16 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 5255 0 0 0 610 6 0 0 25 0 5 0 599977907 395071488 49989 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=385812 CPUtime=6.15 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 16153 0 0 0 612 3 0 0 25 0 5 0 599977907 395071488 49989 33554432000 4194304 4338100 140734566556384 18446744073709551615 4240545 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=385812 CPUtime=6.07 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 9984 0 0 0 604 3 0 0 18 0 5 0 599977907 395071488 49989 33554432000 4194304 4338100 140734566556384 18446744073709551615 260019791316 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.68
Current children cumulated vsize (KiB) 385812

[startup+12.7002 s]
/proc/loadavg: 7.84 7.98 7.62 9/186 21713
/proc/meminfo: memFree=30131796/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=498632 CPUtime=50.17 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 78927 0 0 0 4985 32 0 0 25 0 5 0 599977893 510599168 78300 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 124658 78300 176 36 0 122563 0
[pid=21703/tid=21705] ppid=21701 vsize=498632 CPUtime=12.54 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 23168 0 0 0 1248 6 0 0 25 0 5 0 599977907 510599168 78300 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=498632 CPUtime=12.52 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 8074 0 0 0 1241 11 0 0 18 0 5 0 599977907 510599168 78301 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=498632 CPUtime=12.53 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 28753 0 0 0 1248 5 0 0 25 0 5 0 599977907 510599168 78301 33554432000 4194304 4338100 140734566556384 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=498632 CPUtime=12.4 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 13236 0 0 0 1234 6 0 0 20 0 5 0 599977907 510599168 78301 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 50.17
Current children cumulated vsize (KiB) 498632

[startup+25.5009 s]
/proc/loadavg: 7.94 8.00 7.63 9/186 21713
/proc/meminfo: memFree=29655152/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=652232 CPUtime=101.13 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 117764 0 0 0 10054 59 0 0 25 0 5 0 599977893 667885568 117137 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 163058 117137 176 36 0 160963 0
[pid=21703/tid=21705] ppid=21701 vsize=652232 CPUtime=25.33 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 36763 0 0 0 2522 11 0 0 25 0 5 0 599977907 667885568 117137 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=652232 CPUtime=25.19 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 12874 0 0 0 2503 16 0 0 18 0 5 0 599977907 667885568 117137 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=652232 CPUtime=25.27 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 46921 0 0 0 2517 10 0 0 25 0 5 0 599977907 667885568 117137 33554432000 4194304 4338100 140734566556384 18446744073709551615 260020092446 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=652232 CPUtime=25.19 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 15510 0 0 0 2500 19 0 0 25 0 5 0 599977907 667885568 117137 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 101.13
Current children cumulated vsize (KiB) 652232

[startup+51.1011 s]
/proc/loadavg: 8.02 8.01 7.65 9/186 21714
/proc/meminfo: memFree=28881416/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=949788 CPUtime=203.06 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 193697 0 0 0 20197 109 0 0 25 0 5 0 599977893 972582912 193070 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 237447 193070 176 36 0 235352 0
[pid=21703/tid=21705] ppid=21701 vsize=949788 CPUtime=50.89 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 60225 0 0 0 5069 20 0 0 25 0 5 0 599977907 972582912 193070 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=949788 CPUtime=50.53 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 25045 0 0 0 5029 24 0 0 18 0 5 0 599977907 972582912 193070 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=949788 CPUtime=50.7 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 77141 0 0 0 5055 15 0 0 25 0 5 0 599977907 972582912 193070 33554432000 4194304 4338100 140734566556384 18446744073709551615 4256039 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=949788 CPUtime=50.77 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 25590 0 0 0 5031 46 0 0 25 0 5 0 599977907 972582912 193070 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.06
Current children cumulated vsize (KiB) 949788

[startup+102.307 s]
/proc/loadavg: 8.11 8.04 7.67 9/186 21715
/proc/meminfo: memFree=27887144/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=1359484 CPUtime=407 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 281972 0 0 0 40509 191 0 0 25 0 5 0 599977893 1392111616 281345 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 339871 281345 176 36 0 337776 0
[pid=21703/tid=21705] ppid=21701 vsize=1359484 CPUtime=101.99 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 105911 0 0 0 10158 41 0 0 25 0 5 0 599977907 1392111616 281345 33554432000 4194304 4338100 140734566556384 18446744073709551615 260019790971 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=1359484 CPUtime=101.3 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 33960 0 0 0 10098 32 0 0 18 0 5 0 599977907 1392111616 281345 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=1359484 CPUtime=101.61 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 105196 0 0 0 10140 21 0 0 25 0 5 0 599977907 1392111616 281345 33554432000 4194304 4338100 140734566556384 18446744073709551615 260020092464 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=1359484 CPUtime=101.93 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 31209 0 0 0 10099 94 0 0 20 0 5 0 599977907 1392111616 281345 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 407
Current children cumulated vsize (KiB) 1359484

[startup+162.301 s]
/proc/loadavg: 8.04 8.03 7.69 9/186 21717
/proc/meminfo: memFree=26816856/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=1851448 CPUtime=646.03 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 406942 0 0 0 64328 275 0 0 25 0 5 0 599977893 1895882752 406315 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 462862 406315 176 36 0 460767 0
[pid=21703/tid=21705] ppid=21701 vsize=1851448 CPUtime=161.62 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 168890 0 0 0 16098 64 0 0 25 0 5 0 599977907 1895882752 406315 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=1851448 CPUtime=160.9 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 46990 0 0 0 16049 41 0 0 18 0 5 0 599977907 1895882752 406315 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=1851448 CPUtime=161.46 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 142176 0 0 0 16116 30 0 0 25 0 5 0 599977907 1895882752 406315 33554432000 4194304 4338100 140734566556384 18446744073709551615 260019790945 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=1851448 CPUtime=161.89 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 43190 0 0 0 16052 137 0 0 25 0 5 0 599977907 1895882752 406315 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 646.03
Current children cumulated vsize (KiB) 1851448


################
# More data... #
################

/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 70826 0 0 0 69387 666 0 0 25 0 5 0 599977907 4477444096 1032097 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2797.56
Current children cumulated vsize (KiB) 4372504

[startup+762.306 s]
/proc/loadavg: 8.12 8.08 7.86 9/186 21733
/proc/meminfo: memFree=20826868/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=4592916 CPUtime=3036.79 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 1088684 0 0 0 302202 1477 0 0 25 0 5 0 599977893 4703145984 1088057 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 1148229 1088057 176 36 0 1146134 0
[pid=21703/tid=21705] ppid=21701 vsize=4592916 CPUtime=758.23 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 513136 0 0 0 75623 200 0 0 25 0 5 0 599977907 4703145984 1088057 33554432000 4194304 4338100 140734566556384 18446744073709551615 260019790882 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=4592916 CPUtime=758.18 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 77211 0 0 0 75353 465 0 0 25 0 5 0 599977907 4703145984 1088058 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=4592916 CPUtime=759.83 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 421553 0 0 0 75885 98 0 0 25 0 5 0 599977907 4703145984 1088059 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=4592916 CPUtime=760.41 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 71089 0 0 0 75329 712 0 0 25 0 5 0 599977907 4703145984 1088059 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3036.79
Current children cumulated vsize (KiB) 4592916

[startup+822.301 s]
/proc/loadavg: 8.10 8.08 7.87 9/186 21734
/proc/meminfo: memFree=20477980/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=4766540 CPUtime=3276.02 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 1132689 0 0 0 325970 1632 0 0 25 0 5 0 599977893 4880936960 1132062 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 1191635 1132063 176 36 0 1189540 0
[pid=21703/tid=21705] ppid=21701 vsize=4766540 CPUtime=818.06 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 539298 0 0 0 81595 211 0 0 25 0 5 0 599977907 4880936960 1132063 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=4766540 CPUtime=817.75 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 77312 0 0 0 81238 537 0 0 25 0 5 0 599977907 4880936960 1132063 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=4766540 CPUtime=819.72 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 439189 0 0 0 81871 101 0 0 25 0 5 0 599977907 4880936960 1132063 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=4766540 CPUtime=820.32 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 71195 0 0 0 81252 780 0 0 25 0 5 0 599977907 4880936960 1132063 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3276.02
Current children cumulated vsize (KiB) 4766540

[startup+882.301 s]
/proc/loadavg: 8.03 8.06 7.88 9/186 21736
/proc/meminfo: memFree=20081344/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=4965056 CPUtime=3515.13 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 1183542 0 0 0 349728 1785 0 0 25 0 5 0 599977893 5084217344 1182915 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 1241264 1182915 176 36 0 1239169 0
[pid=21703/tid=21705] ppid=21701 vsize=4965056 CPUtime=877.71 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 572460 0 0 0 87545 226 0 0 25 0 5 0 599977907 5084217344 1182915 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=4965056 CPUtime=877.52 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 77504 0 0 0 87149 603 0 0 25 0 5 0 599977907 5084217344 1182915 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=4965056 CPUtime=879.49 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 456498 0 0 0 87842 107 0 0 25 0 5 0 599977907 5084217344 1182915 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=4965056 CPUtime=880.23 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 71384 0 0 0 87178 845 0 0 25 0 5 0 599977907 5084217344 1182915 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3515.13
Current children cumulated vsize (KiB) 4965056

[startup+942.301 s]
/proc/loadavg: 8.12 8.08 7.90 9/186 21738
/proc/meminfo: memFree=19675556/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=5229656 CPUtime=3754.45 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 1250350 0 0 0 373517 1928 0 0 25 0 5 0 599977893 5355167744 1249723 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 1307414 1249723 176 36 0 1305319 0
[pid=21703/tid=21705] ppid=21701 vsize=5229656 CPUtime=937.36 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 607823 0 0 0 93497 239 0 0 25 0 5 0 599977907 5355167744 1249723 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=5229656 CPUtime=937.36 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 77698 0 0 0 93069 667 0 0 25 0 5 0 599977907 5355167744 1249723 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=5229656 CPUtime=939.39 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 487556 0 0 0 93825 114 0 0 25 0 5 0 599977907 5355167744 1249724 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=5229656 CPUtime=940.17 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 71578 0 0 0 93112 905 0 0 25 0 5 0 599977907 5355167744 1249724 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3754.45
Current children cumulated vsize (KiB) 5229656

[startup+1002.3 s]
/proc/loadavg: 8.04 8.06 7.90 9/186 21739
/proc/meminfo: memFree=19378392/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=5365744 CPUtime=3993.69 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 1286979 0 0 0 397317 2052 0 0 25 0 5 0 599977893 5494521856 1286352 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 1341436 1286352 176 36 0 1339341 0
[pid=21703/tid=21705] ppid=21701 vsize=5365744 CPUtime=996.92 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 620284 0 0 0 99448 244 0 0 25 0 5 0 599977907 5494521856 1286352 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=5365744 CPUtime=997.23 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 78576 0 0 0 98999 724 0 0 25 0 5 0 599977907 5494521856 1286352 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=5365744 CPUtime=999.31 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 510464 0 0 0 99811 120 0 0 25 0 5 0 599977907 5494521856 1286352 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=5365744 CPUtime=1000.06 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 71959 0 0 0 99045 961 0 0 25 0 5 0 599977907 5494521856 1286352 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3993.69
Current children cumulated vsize (KiB) 5365744

[startup+1062.3 s]
/proc/loadavg: 8.17 8.09 7.92 9/186 21741
/proc/meminfo: memFree=19175924/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=5482568 CPUtime=4233.06 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 1307425 0 0 0 421134 2172 0 0 25 0 5 0 599977893 5614149632 1306798 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 1370642 1306798 176 36 0 1368547 0
[pid=21703/tid=21705] ppid=21701 vsize=5482568 CPUtime=1056.52 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 620594 0 0 0 105404 248 0 0 25 0 5 0 599977907 5614149632 1306798 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=5482568 CPUtime=1057.17 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 79443 0 0 0 104937 780 0 0 25 0 5 0 599977907 5614149632 1306798 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=5482568 CPUtime=1059.23 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 529267 0 0 0 105796 127 0 0 25 0 5 0 599977907 5614149632 1306798 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=5482568 CPUtime=1059.96 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 72425 0 0 0 104983 1013 0 0 25 0 5 0 599977907 5614149632 1306798 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4233.06
Current children cumulated vsize (KiB) 5482568

[startup+1122.3 s]
/proc/loadavg: 8.15 8.10 7.93 9/186 21773
/proc/meminfo: memFree=25158004/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=5580592 CPUtime=4472.06 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 1337614 0 0 0 444934 2272 0 0 25 0 5 0 599977893 5714526208 1336987 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 1395148 1336987 176 36 0 1393053 0
[pid=21703/tid=21705] ppid=21701 vsize=5580592 CPUtime=1116.15 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 638868 0 0 0 111358 257 0 0 25 0 5 0 599977907 5714526208 1336987 33554432000 4194304 4338100 140734566556384 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=5580592 CPUtime=1116.94 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 80815 0 0 0 110865 829 0 0 25 0 5 0 599977907 5714526208 1336987 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=5580592 CPUtime=1119.07 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 538458 0 0 0 111778 129 0 0 25 0 5 0 599977907 5714526208 1336987 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=5580592 CPUtime=1119.74 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 73777 0 0 0 110920 1054 0 0 18 0 5 0 599977907 5714526208 1336987 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4472.06
Current children cumulated vsize (KiB) 5580592

[startup+1182.3 s]
/proc/loadavg: 8.10 8.10 7.94 9/186 21775
/proc/meminfo: memFree=24961140/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=5640816 CPUtime=4711.17 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) S 21701 21703 21560 0 -1 4202496 1352929 0 0 0 468807 2310 0 0 25 0 5 0 599977893 5776195584 1352302 33554432000 4194304 4338100 140734566556384 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 1410204 1352302 176 36 0 1408109 0
[pid=21703/tid=21705] ppid=21701 vsize=5640816 CPUtime=1175.73 cores=0,2,4,6
/proc/21703/task/21705/stat : 21705 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 653497 0 0 0 117307 266 0 0 25 0 5 0 599977907 5776195584 1352302 33554432000 4194304 4338100 140734566556384 18446744073709551615 4234354 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21703/tid=21706] ppid=21701 vsize=5640816 CPUtime=1176.85 cores=0,2,4,6
/proc/21703/task/21706/stat : 21706 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 80885 0 0 0 116831 854 0 0 25 0 5 0 599977907 5776195584 1352302 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=5640816 CPUtime=1178.93 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 538729 0 0 0 117763 130 0 0 25 0 5 0 599977907 5776195584 1352302 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21703/tid=21708] ppid=21701 vsize=5640816 CPUtime=1179.5 cores=0,2,4,6
/proc/21703/task/21708/stat : 21708 (ttsth-5-4) R 21701 21703 21560 0 -1 4202560 74122 0 0 0 116893 1057 0 0 18 0 5 0 599977907 5776195584 1352302 33554432000 4194304 4338100 140734566556384 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4711.17
Current children cumulated vsize (KiB) 5640816



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.6 s]
/proc/loadavg: 8.22 8.12 7.95 7/183 21775
/proc/meminfo: memFree=24864396/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=0 CPUtime=4800.05 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) Z 21701 21703 21560 0 -1 4203548 1377808 0 0 0 477676 2329 0 0 18 0 2 0 599977893 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 0 0 0 0 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=0 CPUtime=1201.1 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4203588 547556 0 0 0 119978 132 0 0 25 0 2 0 599977907 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.05
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.8 s]
/proc/loadavg: 8.22 8.12 7.95 7/183 21775
/proc/meminfo: memFree=24864396/32951124 swapFree=67111464/67111528
[pid=21703] ppid=21701 vsize=0 CPUtime=4800.25 cores=0,2,4,6
/proc/21703/stat : 21703 (ttsth-5-4) Z 21701 21703 21560 0 -1 4203548 1377808 0 0 0 477676 2349 0 0 18 0 2 0 599977893 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/21703/statm: 0 0 0 0 0 0 0
[pid=21703/tid=21707] ppid=21701 vsize=0 CPUtime=1201.3 cores=0,2,4,6
/proc/21703/task/21707/stat : 21707 (ttsth-5-4) R 21701 21703 21560 0 -1 4203588 547556 0 0 0 119978 152 0 0 25 0 2 0 599977907 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.83
CPU time (s): 4800.29
CPU user time (s): 4776.77
CPU system time (s): 23.5244
CPU usage (%): 398.419
Max. virtual memory (cumulated for all children) (KiB): 5738312

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4776.77
system time used= 23.5244
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1377808
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 28228
involuntary context switches= 17145

runsolver used 1.91571 second user time and 4.32834 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 19:17:53
IDJOB=3263718
IDBENCH=84010
IDSOLVER=1654
FILE ID=node110/3263718-1304097473
RUNJOBID= node110-1304096259-21578
PBS_JOBID= 13170606
Free space on /tmp= 72836 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/SAT/SGI_30_80_15_90_4-dir.shuffled-as.sat03-6.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263718-1304097473/watcher-3263718-1304097473 -o /tmp/evaluation-result-3263718-1304097473/solver-3263718-1304097473 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263718-1304097473.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 22bbcb4a25a1da66113c240898c75509
RANDOM SEED=1171089639

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30844908 kB
Buffers:        904408 kB
Cached:         697520 kB
SwapCached:         64 kB
Active:         155332 kB
Inactive:      1545056 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30844908 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:            2768 kB
Writeback:           0 kB
AnonPages:       97812 kB
Mapped:          15072 kB
Slab:           341720 kB
PageTables:       4764 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   272172 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72840 MiB
End job on node110 at 2011-04-29 19:38:00