Trace number 3249416

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.76 1202.87

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-21x14.cnf
MD5SUMf76302f2442dfc7cabc2315e61e55fea
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1176
Number of clauses78498
Sum of the clauses size310464
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21764
Number of clauses of size 30
Number of clauses of size 476734
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.08	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249416-1304061863.cnf 
0.00/0.08	c Using SATTIMEOUT of 4800
0.00/0.08	c Using SATRAM of 15500
0.00/0.08	c Size of subset node = 32
0.00/0.08	c Variables = 1176
0.00/0.08	c Clauses = 78498
0.00/0.08	c Literals = 310464
0.00/0.08	c Reduced size:
0.00/0.08	c Variables = 1176
0.00/0.08	c Clauses = 78498
0.00/0.08	c Literals = 310464
0.00/0.08	c a2
0.19/0.23	c a1
0.49/0.50	c Independent sub-problems = 1
0.49/0.50	c Sub-problem with 1176 variables and 78498 clauses
0.49/0.50	c using permutation method 4
0.49/0.50	c count = 1
0.49/0.50	c Ordering finished 0.480 seconds
0.49/0.50	c mapped 0.500 seconds
0.49/0.50	c permuted 0.500 seconds
0.49/0.50	c about to sort
0.49/0.51	c sorted
0.59/0.65	c clauses now 78498
0.59/0.65	c init finished 0.530 seconds
0.59/0.65	c tree built 0.640 seconds
0.59/0.65	c 
0.59/0.65	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.59/0.65	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.59/0.65	c ................................ | ..........                    |                                  0.6         0.0             
0.59/0.65	c ................................ | ..........                  1 |                                  0.6         1.6             
0.59/0.65	c ................................ | ..........                  2 |                                  0.6         3.1             
0.59/0.65	c ................................ | ..........                  3 |                                  0.6         4.7             
0.59/0.65	c ................................ | ..........                  4 |                                  0.6         6.2             
0.59/0.65	c ................................ | ..........                  5 |                                  0.6         7.8             
0.59/0.66	c ................................ | ..........               1000 | 36           477                 0.7      1470.6        13869
95.85/24.54	c ................................ | ..........            1260000 | 132961       359460             96.0     13122.3     28281651
191.69/48.59	c ................................ | ..........            1691000 | 193136       499741            192.1      8804.1     49025468
287.93/72.64	c ................................ | ..........            1943000 | 233440       587191            288.1      6744.4     63335512
384.13/96.71	c ................................ | ..........            2140000 | 268750       664090            384.2      5570.6     74407900
479.98/120.77	c ................................ | *.........            2300000 | 296716       738584            480.3      4789.1     84458108
576.22/144.84	c ................................ | *.........            2445000 | 323379       804021            576.4      4241.9     93324650
672.10/168.88	c ................................ | *.........            2634000 | 357608       869251            672.4      3917.1    102688313
767.94/192.88	c ................................ | *.........            2788000 | 383861       934168            768.3      3628.9    109458527
863.77/216.88	c ................................ | *.........            2936000 | 410500       990328            864.1      3397.7    118783674
960.46/241.07	c ................................ | **........            3064000 | 432889       1043233           960.8      3189.1    127565479
1056.36/265.07	c ................................ | **........            3195000 | 457624       1095999          1056.6      3023.8    135603396
1152.55/289.12	c ................................ | **........            3319000 | 480602       1141490          1152.7      2879.5    141548859
1248.44/313.11	c ................................ | **........            3410000 | 495714       1188971          1248.5      2731.3    146804155
1344.29/337.15	c ................................ | **........            3514000 | 514918       1231922          1344.5      2613.6    153291842
1440.48/361.25	c ................................ | ***.......            3631000 | 538065       1279156          1440.7      2520.3    158251258
1536.35/385.20	c ................................ | ***.......            3760000 | 561791       1326418          1536.4      2447.3    165543775
1632.25/409.21	c ................................ | ***.......            3854000 | 579200       1363738          1632.3      2361.1    170560646
1729.22/433.52	c ................................ | ***.......            3949000 | 596381       1407050          1729.3      2283.5    172259016
1824.29/457.33	c ................................ | ***.......            4049000 | 614361       1447669          1824.4      2219.3    177179202
1920.17/481.32	c ................................ | ****......            4188000 | 636500       1492792          1920.3      2181.0    179693663
2015.88/505.34	c ................................ | ****......            4331000 | 659638       1535240          2016.1      2148.2    180775546
2112.17/529.49	c ................................ | ****......            4489000 | 687262       1584556          2112.6      2124.9    180775546
2208.05/553.42	c ................................ | ****......            4616000 | 711165       1625129          2208.2      2090.4    185112694
2305.11/577.71	c ................................ | ****......            4742000 | 733382       1663173          2305.2      2057.1    189045984
2400.19/601.56	c ................................ | *****.....            4855000 | 751697       1704940          2400.4      2022.5    195010930
2496.48/625.63	c ................................ | *****.....            4963000 | 769545       1748190          2496.6      1987.9    201147186
2591.97/649.57	c ................................ | *****.....            5078000 | 789105       1780997          2592.3      1958.9    205990092
2689.01/673.85	c ................................ | *****.....            5201000 | 811063       1812371          2689.2      1934.0    210536422
2784.11/697.60	c ................................ | *****.....            5297000 | 828591       1844698          2784.1      1902.6    217104499
2880.80/721.83	c ................................ | ******....            5381000 | 842993       1874365          2880.9      1867.8    221318872
2975.80/745.65	c ................................ | ******....            5479000 | 860497       1907096          2976.0      1841.1    223689144
3072.05/769.74	c ................................ | ******....            5586000 | 880199       1936819          3072.2      1818.2    225878638
3167.88/793.76	c ................................ | ******....            5689000 | 897194       1969451          3168.1      1795.7    230749269
3264.17/817.89	c ................................ | ******....            5780000 | 912800       2002883          3264.6      1770.5    236161124
3359.99/841.88	c ................................ | *******...            5889000 | 931458       2037900          3360.3      1752.5    241674757
3457.21/866.28	c ................................ | *******...            5995000 | 948110       2068593          3457.5      1733.9    245428663
3553.51/890.33	c ................................ | *******...            6066000 | 958780       2094035          3553.6      1707.0    246901938
3648.17/914.08	c ................................ | *******...            6167000 | 976569       2122529          3648.5      1690.3    248526846
3745.63/938.43	c ................................ | *******...            6257000 | 992492       2151717          3745.8      1670.4    250793301
3839.90/962.08	c ................................ | ********..            6350000 | 1008987      2180090          3840.2      1653.5    254802877
3936.81/986.31	c ................................ | ********..            6463000 | 1028162      2211200          3936.9      1641.7    255894716
4031.80/1010.18	c ................................ | ********..            6572000 | 1044701      2246896          4032.1      1629.9    255894716
4128.47/1034.34	c ................................ | ********..            6665000 | 1061082      2278643          4128.7      1614.3    255894716
4224.32/1058.32	c ................................ | ********..            6769000 | 1080592      2310349          4224.4      1602.4    255894716
4319.75/1082.26	c ................................ | *********.            6872000 | 1098640      2341496          4320.0      1590.7    255894716
4416.86/1106.56	c ................................ | *********.            6939000 | 1110900      2370100          4417.1      1570.9    255894716
4513.13/1130.64	c ................................ | *********.            7014000 | 1122252      2401654          4513.3      1554.1    255951030
4608.61/1154.52	c ................................ | *********.            7091000 | 1135855      2430800          4608.7      1538.6    257609000
4703.70/1178.38	c ................................ | *********.            7164000 | 1148977      2461086          4704.0      1522.9    259430229

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249416-1304061863/watcher-3249416-1304061863 -o /tmp/evaluation-result-3249416-1304061863/solver-3249416-1304061863 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249416-1304061863.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.76 3.96 3.98 2/174 30387
/proc/meminfo: memFree=28724532/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) R 30385 30387 27559 0 -1 4202496 419 0 0 0 0 0 0 0 22 0 1 0 517787065 9633792 361 33554432000 4194304 4338100 140734878424048 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/30387/statm: 2352 361 147 36 0 257 0

[startup+0.0988631 s]
/proc/loadavg: 3.76 3.96 3.98 2/174 30387
/proc/meminfo: memFree=28724532/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=26816 CPUtime=0.09 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) R 30385 30387 27559 0 -1 4202496 4684 0 0 0 9 0 0 0 22 0 1 0 517787065 27459584 4626 33554432000 4194304 4338100 140734878424048 18446744073709551615 222921473803 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/30387/statm: 6740 4641 151 36 0 4645 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26816

[startup+0.100843 s]
/proc/loadavg: 3.76 3.96 3.98 2/174 30387
/proc/meminfo: memFree=28724532/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=30404 CPUtime=0.09 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) R 30385 30387 27559 0 -1 4202496 5345 0 0 0 9 0 0 0 22 0 1 0 517787065 31133696 5287 33554432000 4194304 4338100 140734878424048 18446744073709551615 222921473803 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/30387/statm: 7637 5301 151 36 0 5542 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 30404

[startup+0.300808 s]
/proc/loadavg: 3.76 3.96 3.98 2/174 30387
/proc/meminfo: memFree=28724532/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=25328 CPUtime=0.29 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) R 30385 30387 27559 0 -1 4202496 7002 0 0 0 29 0 0 0 23 0 1 0 517787065 25935872 4370 33554432000 4194304 4338100 140734878424048 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/30387/statm: 6332 4370 155 36 0 4237 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 25328

[startup+0.700672 s]
/proc/loadavg: 3.76 3.96 3.98 2/174 30387
/proc/meminfo: memFree=28724532/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=229036 CPUtime=0.82 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 12480 0 0 0 81 1 0 0 25 0 5 0 517787065 234532864 9694 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 57259 9694 176 36 0 55164 0
Current children cumulated CPU time (s) 0.82
Current children cumulated vsize (KiB) 229036

[startup+1.50054 s]
/proc/loadavg: 3.62 3.93 3.97 6/182 30398
/proc/meminfo: memFree=28587432/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=273752 CPUtime=4.01 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 25609 0 0 0 398 3 0 0 25 0 5 0 517787065 280322048 22823 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 68438 22823 176 36 0 66343 0
[pid=30387/tid=30395] ppid=30385 vsize=273752 CPUtime=0.82 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 3144 0 0 0 82 0 0 0 19 0 5 0 517787130 280322048 22823 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=273752 CPUtime=0.84 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 3142 0 0 0 84 0 0 0 19 0 5 0 517787130 280322048 22823 33554432000 4194304 4338100 140734878424048 18446744073709551615 222921473465 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=273752 CPUtime=0.84 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 3194 0 0 0 84 0 0 0 25 0 5 0 517787130 280322048 22824 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=273752 CPUtime=0.83 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 4388 0 0 0 83 0 0 0 18 0 5 0 517787130 280322048 22824 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4.01
Current children cumulated vsize (KiB) 273752

[startup+3.10023 s]
/proc/loadavg: 3.62 3.93 3.97 6/182 30398
/proc/meminfo: memFree=28433548/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=420748 CPUtime=10.4 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 45352 0 0 0 1031 9 0 0 25 0 5 0 517787065 430845952 42566 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 105187 42566 176 36 0 103092 0
[pid=30387/tid=30395] ppid=30385 vsize=420748 CPUtime=2.42 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 8303 0 0 0 240 2 0 0 18 0 5 0 517787130 430845952 42566 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=420748 CPUtime=2.43 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 7162 0 0 0 242 1 0 0 22 0 5 0 517787130 430845952 42566 33554432000 4194304 4338100 140734878424048 18446744073709551615 222921172414 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=420748 CPUtime=2.43 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 7577 0 0 0 242 1 0 0 25 0 5 0 517787130 430845952 42566 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=420748 CPUtime=2.43 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 10569 0 0 0 241 2 0 0 19 0 5 0 517787130 430845952 42566 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 10.4
Current children cumulated vsize (KiB) 420748

[startup+6.30066 s]
/proc/loadavg: 3.62 3.93 3.97 6/182 30398
/proc/meminfo: memFree=28294416/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=530756 CPUtime=23.18 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 73047 0 0 0 2302 16 0 0 25 0 5 0 517787065 543494144 70261 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 132689 70261 176 36 0 130594 0
[pid=30387/tid=30395] ppid=30385 vsize=530756 CPUtime=5.6 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 15433 0 0 0 556 4 0 0 25 0 5 0 517787130 543494144 70262 33554432000 4194304 4338100 140734878424048 18446744073709551615 222921172033 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=530756 CPUtime=5.63 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 13284 0 0 0 560 3 0 0 21 0 5 0 517787130 543494144 70262 33554432000 4194304 4338100 140734878424048 18446744073709551615 4286350 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=530756 CPUtime=5.64 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 14993 0 0 0 561 3 0 0 25 0 5 0 517787130 543494144 70262 33554432000 4194304 4338100 140734878424048 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=530756 CPUtime=5.62 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 17597 0 0 0 559 3 0 0 23 0 5 0 517787130 543494144 70262 33554432000 4194304 4338100 140734878424048 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 23.18
Current children cumulated vsize (KiB) 530756

[startup+12.7004 s]
/proc/loadavg: 4.07 4.01 4.00 9/186 30402
/proc/meminfo: memFree=28075348/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=698508 CPUtime=48.72 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 115376 0 0 0 4844 28 0 0 25 0 5 0 517787065 715272192 112590 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 174627 112590 176 36 0 172532 0
[pid=30387/tid=30395] ppid=30385 vsize=698508 CPUtime=11.97 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 23014 0 0 0 1189 8 0 0 20 0 5 0 517787130 715272192 112590 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=698508 CPUtime=12.03 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 21564 0 0 0 1197 6 0 0 25 0 5 0 517787130 715272192 112590 33554432000 4194304 4338100 140734878424048 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=698508 CPUtime=12.03 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 23383 0 0 0 1198 5 0 0 25 0 5 0 517787130 715272192 112590 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=698508 CPUtime=12.02 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 35674 0 0 0 1196 6 0 0 25 0 5 0 517787130 715272192 112590 33554432000 4194304 4338100 140734878424048 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 48.72
Current children cumulated vsize (KiB) 698508

[startup+25.5011 s]
/proc/loadavg: 4.67 4.15 4.04 9/186 30403
/proc/meminfo: memFree=27456788/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=1221912 CPUtime=99.85 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 246142 0 0 0 9933 52 0 0 25 0 5 0 517787065 1251237888 243356 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 305478 243356 176 36 0 303383 0
[pid=30387/tid=30395] ppid=30385 vsize=1221912 CPUtime=24.7 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 42049 0 0 0 2458 12 0 0 25 0 5 0 517787130 1251237888 243356 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=1221912 CPUtime=24.83 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 42240 0 0 0 2473 10 0 0 25 0 5 0 517787130 1251237888 243356 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=1221912 CPUtime=24.83 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 70277 0 0 0 2470 13 0 0 25 0 5 0 517787130 1251237888 243356 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=1221912 CPUtime=24.81 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 79835 0 0 0 2467 14 0 0 25 0 5 0 517787130 1251237888 243356 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 99.85
Current children cumulated vsize (KiB) 1221912

[startup+51.1004 s]
/proc/loadavg: 6.06 4.53 4.17 9/186 30403
/proc/meminfo: memFree=26607040/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=1886576 CPUtime=202.08 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 412414 0 0 0 20125 83 0 0 25 0 5 0 517787065 1931853824 409628 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 471644 409628 176 36 0 469549 0
[pid=30387/tid=30395] ppid=30385 vsize=1886576 CPUtime=50.16 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 86594 0 0 0 4996 20 0 0 25 0 5 0 517787130 1931853824 409628 33554432000 4194304 4338100 140734878424048 18446744073709551615 4288141 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=1886576 CPUtime=50.41 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 75289 0 0 0 5025 16 0 0 25 0 5 0 517787130 1931853824 409628 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=1886576 CPUtime=50.43 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 111118 0 0 0 5022 21 0 0 25 0 5 0 517787130 1931853824 409628 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=1886576 CPUtime=50.4 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 127672 0 0 0 5017 23 0 0 25 0 5 0 517787130 1931853824 409628 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 202.08
Current children cumulated vsize (KiB) 1886576

[startup+102.306 s]
/proc/loadavg: 7.28 5.09 4.38 9/186 30405
/proc/meminfo: memFree=25466064/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=2722544 CPUtime=406.51 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 620261 0 0 0 40530 121 0 0 25 0 5 0 517787065 2787885056 617475 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 680636 617475 176 36 0 678541 0
[pid=30387/tid=30395] ppid=30385 vsize=2722544 CPUtime=101.07 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 143101 0 0 0 10077 30 0 0 25 0 5 0 517787130 2787885056 617475 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=2722544 CPUtime=101.58 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 117002 0 0 0 10135 23 0 0 25 0 5 0 517787130 2787885056 617475 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=2722544 CPUtime=101.6 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 161980 0 0 0 10129 31 0 0 25 0 5 0 517787130 2787885056 617475 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254244 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=2722544 CPUtime=101.57 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 186437 0 0 0 10124 33 0 0 25 0 5 0 517787130 2787885056 617475 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 406.51
Current children cumulated vsize (KiB) 2722544

[startup+162.301 s]
/proc/loadavg: 7.77 5.63 4.61 9/186 30406
/proc/meminfo: memFree=24503164/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=3457888 CPUtime=646.13 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 804449 0 0 0 64454 159 0 0 25 0 5 0 517787065 3540877312 801663 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 864472 801663 176 36 0 862377 0
[pid=30387/tid=30395] ppid=30385 vsize=3457888 CPUtime=160.8 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 189378 0 0 0 16040 40 0 0 25 0 5 0 517787130 3540877312 801663 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=3457888 CPUtime=161.56 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 150949 0 0 0 16126 30 0 0 25 0 5 0 517787130 3540877312 801663 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=3457888 CPUtime=161.58 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 216129 0 0 0 16116 42 0 0 25 0 5 0 517787130 3540877312 801663 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=3457888 CPUtime=161.52 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 236252 0 0 0 16107 45 0 0 25 0 5 0 517787130 3540877312 801663 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 646.13
Current children cumulated vsize (KiB) 3457888


################
# More data... #
################

/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 467356 0 0 0 75977 94 0 0 25 0 5 0 517787130 7674822656 1804301 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3042.5
Current children cumulated vsize (KiB) 7494944

[startup+822.3 s]
/proc/loadavg: 8.01 7.78 6.35 9/186 30424
/proc/meminfo: memFree=17980336/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=7867440 CPUtime=3282.11 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 1900802 0 0 0 327821 390 0 0 25 0 5 0 517787065 8056258560 1898016 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 1966860 1898016 176 36 0 1964765 0
[pid=30387/tid=30395] ppid=30385 vsize=7867440 CPUtime=820.17 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 530811 0 0 0 81904 113 0 0 25 0 5 0 517787130 8056258560 1898016 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=7867440 CPUtime=821.2 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 355695 0 0 0 82048 72 0 0 25 0 5 0 517787130 8056258560 1898016 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=7867440 CPUtime=819.36 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 512414 0 0 0 81832 104 0 0 25 0 5 0 517787130 8056258560 1898016 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=7867440 CPUtime=820.69 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 490141 0 0 0 81971 98 0 0 25 0 5 0 517787130 8056258560 1898016 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3282.11
Current children cumulated vsize (KiB) 7867440

[startup+882.3 s]
/proc/loadavg: 8.00 7.82 6.45 9/186 30425
/proc/meminfo: memFree=17545064/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=8172832 CPUtime=3521.54 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 1974800 0 0 0 351748 406 0 0 25 0 5 0 517787065 8368979968 1972014 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 2043208 1972014 176 36 0 2041113 0
[pid=30387/tid=30395] ppid=30385 vsize=8172832 CPUtime=880.09 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 551382 0 0 0 87892 117 0 0 25 0 5 0 517787130 8368979968 1972014 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=8172832 CPUtime=881.19 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 368602 0 0 0 88045 74 0 0 25 0 5 0 517787130 8368979968 1972014 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=8172832 CPUtime=878.98 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 531299 0 0 0 87790 108 0 0 25 0 5 0 517787130 8368979968 1972014 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=8172832 CPUtime=880.58 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 511776 0 0 0 87955 103 0 0 25 0 5 0 517787130 8368979968 1972014 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3521.54
Current children cumulated vsize (KiB) 8172832

[startup+942.3 s]
/proc/loadavg: 8.04 7.86 6.55 9/186 30427
/proc/meminfo: memFree=17300116/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=8335752 CPUtime=3761.21 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 2016040 0 0 0 375707 414 0 0 25 0 5 0 517787065 8535810048 2013254 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 2083938 2013254 176 36 0 2081843 0
[pid=30387/tid=30395] ppid=30385 vsize=8335752 CPUtime=940.09 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 564998 0 0 0 93889 120 0 0 25 0 5 0 517787130 8535810048 2013254 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=8335752 CPUtime=941.17 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 368632 0 0 0 94043 74 0 0 25 0 5 0 517787130 8535810048 2013254 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=8335752 CPUtime=938.69 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 545919 0 0 0 93758 111 0 0 25 0 5 0 517787130 8535810048 2013254 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=8335752 CPUtime=940.57 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 524750 0 0 0 93952 105 0 0 25 0 5 0 517787130 8535810048 2013254 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3761.21
Current children cumulated vsize (KiB) 8335752

[startup+1002.3 s]
/proc/loadavg: 8.13 7.92 6.65 9/186 30429
/proc/meminfo: memFree=17047328/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=8567948 CPUtime=4000.72 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 2064692 0 0 0 399646 426 0 0 25 0 5 0 517787065 8773578752 2061906 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 2141987 2061906 176 36 0 2139892 0
[pid=30387/tid=30395] ppid=30385 vsize=8567948 CPUtime=1000.04 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 585587 0 0 0 99879 125 0 0 25 0 5 0 517787130 8773578752 2061906 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=8567948 CPUtime=1001.15 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 376359 0 0 0 100039 76 0 0 25 0 5 0 517787130 8773578752 2061906 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=8567948 CPUtime=998.38 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 551382 0 0 0 99726 112 0 0 25 0 5 0 517787130 8773578752 2061906 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=8567948 CPUtime=1000.46 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 539623 0 0 0 99937 109 0 0 25 0 5 0 517787130 8773578752 2061906 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4000.72
Current children cumulated vsize (KiB) 8567948

[startup+1062.3 s]
/proc/loadavg: 8.12 7.94 6.74 9/186 30430
/proc/meminfo: memFree=16941744/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=8609172 CPUtime=4240.24 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 2076464 0 0 0 423594 430 0 0 25 0 5 0 517787065 8815792128 2073678 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 2152293 2073678 176 36 0 2150198 0
[pid=30387/tid=30395] ppid=30385 vsize=8609172 CPUtime=1059.95 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 586507 0 0 0 105869 126 0 0 25 0 5 0 517787130 8815792128 2073678 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=8609172 CPUtime=1061.05 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 378585 0 0 0 106028 77 0 0 25 0 5 0 517787130 8815792128 2073678 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=8609172 CPUtime=1058.1 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 556006 0 0 0 105696 114 0 0 25 0 5 0 517787130 8815792128 2073678 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=8609172 CPUtime=1060.45 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 543625 0 0 0 105935 110 0 0 25 0 5 0 517787130 8815792128 2073678 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4240.24
Current children cumulated vsize (KiB) 8609172

[startup+1122.3 s]
/proc/loadavg: 8.14 7.98 6.82 9/186 30432
/proc/meminfo: memFree=16818920/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=8609304 CPUtime=4479.96 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 2076630 0 0 0 447565 431 0 0 25 0 5 0 517787065 8815927296 2073844 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 2152326 2073844 176 36 0 2150231 0
[pid=30387/tid=30395] ppid=30385 vsize=8609304 CPUtime=1119.96 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 586538 0 0 0 111869 127 0 0 25 0 5 0 517787130 8815927296 2073844 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=8609304 CPUtime=1121.05 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 378621 0 0 0 112028 77 0 0 25 0 5 0 517787130 8815927296 2073844 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=8609304 CPUtime=1117.83 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 556059 0 0 0 111668 115 0 0 25 0 5 0 517787130 8815927296 2073844 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=8609304 CPUtime=1120.45 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 543671 0 0 0 111935 110 0 0 25 0 5 0 517787130 8815927296 2073844 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4479.96
Current children cumulated vsize (KiB) 8609304

[startup+1182.3 s]
/proc/loadavg: 8.19 8.02 6.90 9/186 30433
/proc/meminfo: memFree=16507376/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=8731560 CPUtime=4719.68 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 2107099 0 0 0 471530 438 0 0 25 0 5 0 517787065 8941117440 2104313 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 2182890 2104313 176 36 0 2180795 0
[pid=30387/tid=30395] ppid=30385 vsize=8731560 CPUtime=1179.94 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 586569 0 0 0 117867 127 0 0 25 0 5 0 517787130 8941117440 2104313 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=8731560 CPUtime=1181.04 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 378656 0 0 0 118027 77 0 0 25 0 5 0 517787130 8941117440 2104313 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=8731560 CPUtime=1177.56 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 578403 0 0 0 117636 120 0 0 25 0 5 0 517787130 8941117440 2104313 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=8731560 CPUtime=1180.46 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 551730 0 0 0 117935 111 0 0 25 0 5 0 517787130 8941117440 2104313 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4719.68
Current children cumulated vsize (KiB) 8731560



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.5 s]
/proc/loadavg: 8.14 8.01 6.93 9/186 30434
/proc/meminfo: memFree=16324696/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=8816016 CPUtime=4800.38 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) S 30385 30387 27559 0 -1 4202496 2129536 0 0 0 479596 442 0 0 25 0 5 0 517787065 9027600384 2126750 33554432000 4194304 4338100 140734878424048 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 2204004 2126750 176 36 0 2201909 0
[pid=30387/tid=30395] ppid=30385 vsize=8816016 CPUtime=1200.14 cores=0,2,4,6
/proc/30387/task/30395/stat : 30395 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 589288 0 0 0 119887 127 0 0 25 0 5 0 517787130 9027600384 2126750 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30387/tid=30396] ppid=30385 vsize=8816016 CPUtime=1201.25 cores=0,2,4,6
/proc/30387/task/30396/stat : 30396 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 381515 0 0 0 120047 78 0 0 25 0 5 0 517787130 9027600384 2126750 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=8816016 CPUtime=1197.67 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 588343 0 0 0 119645 122 0 0 25 0 5 0 517787130 9027600384 2126750 33554432000 4194304 4338100 140734878424048 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30387/tid=30398] ppid=30385 vsize=8816016 CPUtime=1200.65 cores=0,2,4,6
/proc/30387/task/30398/stat : 30398 (ttsth-5-4) R 30385 30387 27559 0 -1 4202560 558649 0 0 0 119953 112 0 0 25 0 5 0 517787130 9027600384 2126750 33554432000 4194304 4338100 140734878424048 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.38
Current children cumulated vsize (KiB) 8816016

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.61 s]
/proc/loadavg: 8.14 8.01 6.93 9/186 30434
/proc/meminfo: memFree=16324696/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=0 CPUtime=4800.5 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) Z 30385 30387 27559 0 -1 4203548 2129541 0 0 0 479598 452 0 0 18 0 2 0 517787065 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 0 0 0 0 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=0 CPUtime=1197.77 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4203588 588345 0 0 0 119645 132 0 0 25 0 2 0 517787130 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.5
Current children cumulated vsize (KiB) 0

[startup+1202.8 s]
/proc/loadavg: 8.14 8.01 6.93 9/186 30434
/proc/meminfo: memFree=16324696/32951124 swapFree=67111356/67111528
[pid=30387] ppid=30385 vsize=0 CPUtime=4800.69 cores=0,2,4,6
/proc/30387/stat : 30387 (ttsth-5-4) Z 30385 30387 27559 0 -1 4203548 2129541 0 0 0 479598 471 0 0 18 0 2 0 517787065 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/30387/statm: 0 0 0 0 0 0 0
[pid=30387/tid=30397] ppid=30385 vsize=0 CPUtime=1197.95 cores=0,2,4,6
/proc/30387/task/30397/stat : 30397 (ttsth-5-4) R 30385 30387 27559 0 -1 4203588 588345 0 0 0 119645 150 0 0 25 0 2 0 517787130 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.69
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.87
CPU time (s): 4800.76
CPU user time (s): 4795.99
CPU system time (s): 4.77627
CPU usage (%): 399.111
Max. virtual memory (cumulated for all children) (KiB): 8816016

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.99
system time used= 4.77627
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2129541
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2154
involuntary context switches= 23330

runsolver used 1.53677 second user time and 4.06138 second system time

The end

Launcher Data 

Begin job on node109 at 2011-04-29 09:24:23
IDJOB=3249416
IDBENCH=83167
IDSOLVER=1654
FILE ID=node109/3249416-1304061863
RUNJOBID= node109-1304061863-30373
PBS_JOBID= 13170356
Free space on /tmp= 72380 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-21x14.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249416-1304061863/watcher-3249416-1304061863 -o /tmp/evaluation-result-3249416-1304061863/solver-3249416-1304061863 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249416-1304061863.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f76302f2442dfc7cabc2315e61e55fea
RANDOM SEED=650058237

node109.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28725076 kB
Buffers:        884244 kB
Cached:        2848936 kB
SwapCached:        168 kB
Active:         821456 kB
Inactive:      2979464 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28725076 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            4292 kB
Writeback:           0 kB
AnonPages:       68084 kB
Mapped:          14500 kB
Slab:           360756 kB
PageTables:       4728 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   193400 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72368 MiB
End job on node109 at 2011-04-29 09:44:28