Trace number 3251704

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.43 1203.97

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m42-q585-n40-p30-s19690873.cnf
MD5SUMe096925b96a6999132833f44e4f216d7
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1.3388
Satisfiable
(Un)Satisfiability was proved
Number of variables1638
Number of clauses253275
Sum of the clauses size508108
Maximum clause length42
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2253235
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.09/0.17	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251704-1304108637.cnf 
0.09/0.17	c Using SATTIMEOUT of 4800
0.09/0.17	c Using SATRAM of 15500
0.09/0.17	c Size of subset node = 32
0.09/0.17	c Variables = 1638
0.09/0.17	c Clauses = 253275
0.09/0.17	c Literals = 508108
0.09/0.17	c Reduced size:
0.09/0.17	c Variables = 1638
0.09/0.17	c Clauses = 253275
0.09/0.17	c Literals = 508108
0.09/0.17	c a2
0.29/0.31	c a1
0.69/0.71	c Independent sub-problems = 1
0.69/0.71	c Sub-problem with 1638 variables and 253275 clauses
0.69/0.71	c using permutation method 4
0.69/0.71	c count = 1
0.69/0.71	c Ordering finished 0.670 seconds
0.69/0.71	c mapped 0.700 seconds
0.69/0.71	c permuted 0.700 seconds
0.69/0.71	c about to sort
0.69/0.73	c sorted
1.39/1.48	c clauses now 253275
1.39/1.48	c init finished 0.780 seconds
1.39/1.48	c tree built 1.480 seconds
1.39/1.48	c 
1.39/1.48	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.39/1.48	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.39/1.48	c ................................ | ..........                    |                                  1.5         0.0             
1.39/1.48	c ................................ | ..........                  1 |                                  1.5         0.7             
1.39/1.48	c ................................ | ..........                  2 |                                  1.5         1.4             
1.39/1.48	c ................................ | ..........                  3 |                                  1.5         2.0             
1.39/1.48	c ................................ | ..........                  4 |                                  1.5         2.7             
1.39/1.48	c ................................ | ..........                  5 |                                  1.5         3.4             
1.39/1.49	c ................................ | ..........               1000 |              63                  1.5       671.1         1197
95.79/25.15	c ................................ | ..........            5491000 | 1922         580311             96.0     57197.9     20412078
192.01/49.20	c ................................ | ..........           10515000 | 3803         1117640           192.0     54762.8     35081535
287.84/73.24	c ................................ | ..........           15472000 | 5142         1639251           288.0     53722.2     47727965
383.69/97.28	c ................................ | ..........           20470000 | 6949         2147018           384.0     53305.9     68568591
479.92/121.32	c ................................ | *.........           25134000 | 8850         2629888           480.0     52362.5     78110081
575.73/145.37	c ................................ | *.........           29488000 | 10460        3092152           576.0     51194.4     81866556
671.98/169.41	c ................................ | *.........           33841000 | 11942        3548138           672.0     50357.9     91609716
767.81/193.45	c ................................ | *.........           38137000 | 13811        3988777           768.0     49656.9     97732921
863.62/217.49	c ................................ | *.........           42380000 | 16235        4431940           864.0     49050.9    107726447
959.81/241.54	c ................................ | **........           46608000 | 17972        4862998           960.0     48549.5    118027715
1055.62/265.59	c ................................ | **........           50641000 | 19511        5305562          1056.0     47955.0    127174155
1151.85/289.64	c ................................ | **........           54718000 | 21314        5741089          1152.0     47497.9    141647367
1247.65/313.68	c ................................ | **........           58376000 | 22977        6147776          1248.0     46775.6    145299159
1343.94/337.72	c ................................ | **........           62210000 | 24837        6571469          1344.0     46286.9    152059864
1439.73/361.76	c ................................ | ***.......           65965000 | 26073        6976148          1440.0     45809.0    160137423
1535.92/385.82	c ................................ | ***.......           69497000 | 27532        7398233          1536.0     45245.4    165882021
1631.71/409.87	c ................................ | ***.......           72994000 | 28794        7826787          1632.0     44726.4    174198055
1727.93/433.91	c ................................ | ***.......           77644000 | 32036        8276766          1728.0     44932.9    182677641
1823.73/457.96	c ................................ | ***.......           82065000 | 34971        8707305          1824.0     44991.5    188005801
1919.87/482.03	c ................................ | ****......           86210000 | 38056        9131756          1920.0     44901.0    195561092
2015.65/506.09	c ................................ | ****......           90549000 | 41314        9542762          2016.0     44914.7    200740629
2111.87/530.14	c ................................ | ****......           94824000 | 44437        9944529          2112.0     44897.5    204197413
2207.67/554.18	c ................................ | ****......           99033000 | 47120        10364932         2208.0     44851.7    210574100
2303.88/578.23	c ................................ | ****......          103168000 | 49945        10779512         2304.0     44777.8    218428674
2399.64/602.29	c ................................ | *****.....          107184000 | 52647        11185323         2400.0     44659.8    224524933
2495.83/626.34	c ................................ | *****.....          111266000 | 55438        11585532         2496.0     44577.5    230774147
2591.62/650.40	c ................................ | *****.....          115148000 | 58322        11977362         2592.0     44424.2    236944200
2687.84/674.44	c ................................ | *****.....          119088000 | 60620        12368440         2688.0     44303.2    245294681
2783.64/698.50	c ................................ | *****.....          122877000 | 63905        12742901         2784.0     44136.7    251106975
2879.79/722.55	c ................................ | ******....          126616000 | 67668        13114740         2880.0     43963.9    257671551
2976.01/746.60	c ................................ | ******....          130430000 | 70489        13489091         2976.0     43827.1    265870071
3071.83/770.64	c ................................ | ******....          134062000 | 73485        13862646         3072.0     43639.8    269641704
3167.65/794.69	c ................................ | ******....          137649000 | 77212        14227474         3168.0     43449.7    273125527
3263.87/818.73	c ................................ | ******....          141261000 | 80399        14613408         3264.0     43278.4    279995231
3359.67/842.78	c ................................ | *******...          145065000 | 84990        15032286         3360.0     43174.1    290138474
3455.91/866.83	c ................................ | *******...          148598000 | 87841        15435035         3456.0     42997.1    297647001
3551.66/890.88	c ................................ | *******...          151871000 | 90658        15816864         3552.0     42756.2    299884996
3647.88/914.93	c ................................ | *******...          155204000 | 93560        16209250         3648.0     42544.7    306646194
3743.96/939.01	c ................................ | *******...          159346000 | 97441        16600003         3744.0     42560.2    312987194
3839.71/963.07	c ................................ | ********..          163617000 | 103922       16990530         3840.0     42608.6    316201304
3935.93/987.12	c ................................ | ********..          167706000 | 110293       17366836         3936.0     42608.1    319060507
4031.72/1011.17	c ................................ | ********..          171625000 | 115441       17731202         4032.0     42565.7    321963301
4127.88/1035.23	c ................................ | ********..          175494000 | 120040       18086593         4128.0     42512.9    321963301
4223.71/1059.27	c ................................ | ********..          179681000 | 125695       18484100         4224.0     42538.1    326430644
4319.92/1083.32	c ................................ | *********.          183804000 | 130861       18879865         4320.0     42547.1    332607447
4415.74/1107.36	c ................................ | *********.          187839000 | 136994       19271886         4416.0     42535.9    340840376
4511.98/1131.41	c ................................ | *********.          191544000 | 141963       19635039         4512.0     42452.0    343436078
4607.78/1155.45	c ................................ | *********.          195373000 | 148278       20003645         4608.0     42398.6    348673702
4703.63/1179.49	c ................................ | *********.          199208000 | 156697       20365839         4704.0     42348.6    355929359
4799.77/1203.56	c CPU limit (4800) reached
4799.77/1203.56	c Time = 4800.02 seconds
4799.77/1203.56	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251704-1304108637/watcher-3251704-1304108637 -o /tmp/evaluation-result-3251704-1304108637/solver-3251704-1304108637 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251704-1304108637.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.13 8.06 8.01 5/179 26789
/proc/meminfo: memFree=27396468/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=9764 CPUtime=0 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) R 26787 26789 26305 0 -1 4202496 489 0 0 0 0 0 0 0 25 0 1 0 522463394 9998336 431 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664808192 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/26789/statm: 2441 431 147 36 0 346 0

[startup+0.088128 s]
/proc/loadavg: 8.13 8.06 8.01 5/179 26789
/proc/meminfo: memFree=27396468/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=26568 CPUtime=0.08 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) R 26787 26789 26305 0 -1 4202496 4599 0 0 0 8 0 0 0 25 0 1 0 522463394 27205632 4541 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/26789/statm: 6642 4544 148 36 0 4547 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 26568

[startup+0.101136 s]
/proc/loadavg: 8.13 8.06 8.01 5/179 26789
/proc/meminfo: memFree=27396468/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=28944 CPUtime=0.09 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) R 26787 26789 26305 0 -1 4202496 5248 0 0 0 9 0 0 0 25 0 1 0 522463394 29638656 5190 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/26789/statm: 7236 5191 148 36 0 5141 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28944

[startup+0.301105 s]
/proc/loadavg: 8.13 8.06 8.01 5/179 26789
/proc/meminfo: memFree=27396468/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=58740 CPUtime=0.29 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) R 26787 26789 26305 0 -1 4202496 12113 0 0 0 29 0 0 0 25 0 1 0 522463394 60149760 12054 33554432000 4194304 4338100 140734746055488 18446744073709551615 4206990 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/26789/statm: 14685 12054 153 36 0 12590 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 58740

[startup+0.701038 s]
/proc/loadavg: 8.13 8.06 8.01 5/179 26789
/proc/meminfo: memFree=27396468/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=65316 CPUtime=0.69 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) R 26787 26789 26305 0 -1 4202496 18120 0 0 0 67 2 0 0 25 0 1 0 522463394 66883584 14363 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664204593 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/26789/statm: 16329 14363 161 36 0 14234 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 65316

[startup+1.50085 s]
/proc/loadavg: 8.13 8.06 8.01 6/180 26790
/proc/meminfo: memFree=27311300/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=268204 CPUtime=1.53 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 40423 0 0 0 148 5 0 0 25 0 5 0 522463394 274640896 35676 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 67051 35677 176 36 0 64956 0
Current children cumulated CPU time (s) 1.53
Current children cumulated vsize (KiB) 268204

[startup+3.10057 s]
/proc/loadavg: 8.13 8.06 8.01 9/184 26794
/proc/meminfo: memFree=27217556/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=329484 CPUtime=7.91 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 57672 0 0 0 768 23 0 0 25 0 5 0 522463394 337391616 52925 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 82371 52926 176 36 0 80276 0
[pid=26789/tid=26791] ppid=26787 vsize=329484 CPUtime=1.6 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 5664 0 0 0 159 1 0 0 20 0 5 0 522463542 337391616 52926 33554432000 4194304 4338100 140734746055488 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=329484 CPUtime=1.59 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 2839 0 0 0 152 7 0 0 20 0 5 0 522463542 337391616 52926 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203387 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=329484 CPUtime=1.6 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 6210 0 0 0 160 0 0 0 20 0 5 0 522463542 337391616 52926 33554432000 4194304 4338100 140734746055488 18446744073709551615 4245192 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=329484 CPUtime=1.59 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 3128 0 0 0 152 7 0 0 25 0 5 0 522463542 337391616 52926 33554432000 4194304 4338100 140734746055488 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 7.91
Current children cumulated vsize (KiB) 329484

[startup+6.30102 s]
/proc/loadavg: 8.12 8.06 8.01 13/184 26794
/proc/meminfo: memFree=27054376/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=450164 CPUtime=20.7 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 86524 0 0 0 2005 65 0 0 25 0 5 0 522463394 460967936 81777 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 112541 81778 176 36 0 110446 0
[pid=26789/tid=26791] ppid=26787 vsize=450164 CPUtime=4.81 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 15547 0 0 0 477 4 0 0 25 0 5 0 522463542 460967936 81778 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203387 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=450164 CPUtime=4.78 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 6668 0 0 0 453 25 0 0 25 0 5 0 522463542 460967936 81778 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=450164 CPUtime=4.8 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 16848 0 0 0 478 2 0 0 25 0 5 0 522463542 460967936 81778 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664466176 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=450164 CPUtime=4.8 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 7630 0 0 0 452 28 0 0 25 0 5 0 522463542 460967936 81779 33554432000 4194304 4338100 140734746055488 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 20.7
Current children cumulated vsize (KiB) 450164

[startup+12.7009 s]
/proc/loadavg: 8.11 8.05 8.01 9/184 26794
/proc/meminfo: memFree=26747532/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=734660 CPUtime=46.26 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 142244 0 0 0 4472 154 0 0 25 0 5 0 522463394 752291840 137497 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 183665 137497 176 36 0 181570 0
[pid=26789/tid=26791] ppid=26787 vsize=734660 CPUtime=11.21 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 35857 0 0 0 1112 9 0 0 25 0 5 0 522463542 752291840 137497 33554432000 4194304 4338100 140734746055488 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=734660 CPUtime=11.14 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 15775 0 0 0 1049 65 0 0 25 0 5 0 522463542 752291840 137497 33554432000 4194304 4338100 140734746055488 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=734660 CPUtime=11.2 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 34641 0 0 0 1115 5 0 0 25 0 5 0 522463542 752291840 137498 33554432000 4194304 4338100 140734746055488 18446744073709551615 4254041 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=734660 CPUtime=11.19 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 16139 0 0 0 1051 68 0 0 25 0 5 0 522463542 752291840 137498 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664204002 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 46.26
Current children cumulated vsize (KiB) 734660

[startup+25.5007 s]
/proc/loadavg: 8.15 8.07 8.02 9/184 26795
/proc/meminfo: memFree=26335356/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=1007456 CPUtime=97.38 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 208494 0 0 0 9405 333 0 0 25 0 5 0 522463394 1031634944 202499 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 251864 202499 176 36 0 249769 0
[pid=26789/tid=26791] ppid=26787 vsize=1007456 CPUtime=24 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 64561 0 0 0 2383 17 0 0 25 0 5 0 522463542 1031634944 202499 33554432000 4194304 4338100 140734746055488 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=1007456 CPUtime=23.86 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 26575 0 0 0 2239 147 0 0 25 0 5 0 522463542 1031634944 202499 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676633976 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=1007456 CPUtime=24 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 55480 0 0 0 2390 10 0 0 25 0 5 0 522463542 1031634944 202499 33554432000 4194304 4338100 140734746055488 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=1007456 CPUtime=23.99 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 22046 0 0 0 2247 152 0 0 25 0 5 0 522463542 1031634944 202499 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676621006 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 97.38
Current children cumulated vsize (KiB) 1007456

[startup+51.1003 s]
/proc/loadavg: 8.10 8.06 8.01 9/184 26795
/proc/meminfo: memFree=25601800/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=1477516 CPUtime=199.59 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 326758 0 0 0 19271 688 0 0 25 0 5 0 522463394 1512976384 320763 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 369379 320763 176 36 0 367284 0
[pid=26789/tid=26791] ppid=26787 vsize=1477516 CPUtime=49.6 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 119212 0 0 0 4927 33 0 0 25 0 5 0 522463542 1512976384 320763 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203692 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=1477516 CPUtime=49.33 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 39113 0 0 0 4622 311 0 0 25 0 5 0 522463542 1512976384 320763 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203489 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=1477516 CPUtime=49.61 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 95931 0 0 0 4942 19 0 0 25 0 5 0 522463542 1512976384 320763 33554432000 4194304 4338100 140734746055488 18446744073709551615 4283611 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=1477516 CPUtime=49.54 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 32670 0 0 0 4636 318 0 0 25 0 5 0 522463542 1512976384 320763 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203645 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 199.59
Current children cumulated vsize (KiB) 1477516

[startup+102.306 s]
/proc/loadavg: 8.13 8.08 8.02 9/184 26797
/proc/meminfo: memFree=23889456/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=2666140 CPUtime=404.07 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 620972 0 0 0 39015 1392 0 0 25 0 5 0 522463394 2730127360 614977 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 666535 614977 176 36 0 664440 0
[pid=26789/tid=26791] ppid=26787 vsize=2666140 CPUtime=100.8 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 256093 0 0 0 10013 67 0 0 25 0 5 0 522463542 2730127360 614977 33554432000 4194304 4338100 140734746055488 18446744073709551615 4229545 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=2666140 CPUtime=100.3 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 75603 0 0 0 9388 642 0 0 25 0 5 0 522463542 2730127360 614977 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203706 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=2666140 CPUtime=100.8 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 187207 0 0 0 10046 34 0 0 25 0 5 0 522463542 2730127360 614977 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676622491 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=2666140 CPUtime=100.65 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 62237 0 0 0 9423 642 0 0 25 0 5 0 522463542 2730127360 614977 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664504875 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 404.07
Current children cumulated vsize (KiB) 2666140

[startup+162.301 s]
/proc/loadavg: 8.06 8.06 8.01 9/184 26798
/proc/meminfo: memFree=22591496/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=3322032 CPUtime=643.61 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 780200 0 0 0 62187 2174 0 0 25 0 5 0 522463394 3401760768 774205 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 830508 774205 176 36 0 828413 0
[pid=26789/tid=26791] ppid=26787 vsize=3322032 CPUtime=160.78 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 356100 0 0 0 15980 98 0 0 25 0 5 0 522463542 3401760768 774205 33554432000 4194304 4338100 140734746055488 18446744073709551615 4235674 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=3322032 CPUtime=160.2 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 81393 0 0 0 15009 1011 0 0 25 0 5 0 522463542 3401760768 774205 33554432000 4194304 4338100 140734746055488 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=3322032 CPUtime=160.79 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 235940 0 0 0 16035 44 0 0 25 0 5 0 522463542 3401760768 774205 33554432000 4194304 4338100 140734746055488 18446744073709551615 4235661 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=3322032 CPUtime=160.32 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 66935 0 0 0 15018 1014 0 0 25 0 5 0 522463542 3401760768 774205 33554432000 4194304 4338100 140734746055488 18446744073709551615 4234192 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 643.61
Current children cumulated vsize (KiB) 3322032

[startup+222.301 s]
/proc/loadavg: 7.89 8.02 8.00 9/184 26800
/proc/meminfo: memFree=21203776/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=3907956 CPUtime=883.18 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 932107 0 0 0 85358 2960 0 0 25 0 5 0 522463394 4001746944 926112 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 976989 926113 176 36 0 974894 0
[pid=26789/tid=26791] ppid=26787 vsize=3907956 CPUtime=220.76 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 451547 0 0 0 21946 130 0 0 25 0 5 0 522463542 4001746944 926113 33554432000 4194304 4338100 140734746055488 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0

################
# More data... #
################

/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 1255248 0 0 0 75484 408 0 0 25 0 5 0 522463542 9389178880 2203560 33554432000 4194304 4338100 140734746055488 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=9169120 CPUtime=758.64 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 120270 0 0 0 71271 4593 0 0 25 0 5 0 522463542 9389178880 2203560 33554432000 4194304 4338100 140734746055488 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=9169120 CPUtime=760.32 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 696337 0 0 0 75904 128 0 0 25 0 5 0 522463542 9389178880 2203560 33554432000 4194304 4338100 140734746055488 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=9169120 CPUtime=759.32 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 97868 0 0 0 71325 4607 0 0 25 0 5 0 522463542 9389178880 2203560 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203329 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3038.71
Current children cumulated vsize (KiB) 9169120

[startup+822.301 s]
/proc/loadavg: 7.96 8.00 7.99 9/184 26816
/proc/meminfo: memFree=11672832/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=9582148 CPUtime=3278.24 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 2325111 0 0 0 317413 10411 0 0 25 0 5 0 522463394 9812119552 2319116 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 2395537 2319116 176 36 0 2393442 0
[pid=26789/tid=26791] ppid=26787 vsize=9582148 CPUtime=818.85 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 1312689 0 0 0 81456 429 0 0 25 0 5 0 522463542 9812119552 2319116 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203401 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=9582148 CPUtime=818.35 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 126959 0 0 0 76928 4907 0 0 25 0 5 0 522463542 9812119552 2319117 33554432000 4194304 4338100 140734746055488 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=9582148 CPUtime=820.29 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 741913 0 0 0 81892 137 0 0 25 0 5 0 522463542 9812119552 2319117 33554432000 4194304 4338100 140734746055488 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=9582148 CPUtime=819.22 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 103719 0 0 0 76991 4931 0 0 25 0 5 0 522463542 9812119552 2319117 33554432000 4194304 4338100 140734746055488 18446744073709551615 4255920 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3278.24
Current children cumulated vsize (KiB) 9582148

[startup+882.3 s]
/proc/loadavg: 8.03 8.01 8.00 9/184 26817
/proc/meminfo: memFree=10685124/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=10123292 CPUtime=3517.75 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 2459949 0 0 0 340584 11191 0 0 25 0 5 0 522463394 10366251008 2453954 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 2530823 2453954 176 36 0 2528728 0
[pid=26789/tid=26791] ppid=26787 vsize=10123292 CPUtime=878.8 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 1405877 0 0 0 87422 458 0 0 25 0 5 0 522463542 10366251008 2453954 33554432000 4194304 4338100 140734746055488 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=10123292 CPUtime=878.23 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 126963 0 0 0 82538 5285 0 0 25 0 5 0 522463542 10366251008 2453954 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203745 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=10123292 CPUtime=880.25 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 783554 0 0 0 87881 144 0 0 25 0 5 0 522463542 10366251008 2453954 33554432000 4194304 4338100 140734746055488 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=10123292 CPUtime=878.96 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 103723 0 0 0 82598 5298 0 0 25 0 5 0 522463542 10366251008 2453954 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203689 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3517.75
Current children cumulated vsize (KiB) 10123292

[startup+942.301 s]
/proc/loadavg: 8.05 8.02 8.00 9/184 26819
/proc/meminfo: memFree=10029988/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=10589496 CPUtime=3757.13 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 2581892 0 0 0 363768 11945 0 0 25 0 5 0 522463394 10843643904 2575897 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 2647374 2575897 176 36 0 2645279 0
[pid=26789/tid=26791] ppid=26787 vsize=10589496 CPUtime=938.71 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 1481674 0 0 0 93385 486 0 0 25 0 5 0 522463542 10843643904 2575897 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664204014 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=10589496 CPUtime=937.8 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 129775 0 0 0 88138 5642 0 0 25 0 5 0 522463542 10843643904 2575897 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=10589496 CPUtime=940.2 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 824489 0 0 0 93868 152 0 0 25 0 5 0 522463542 10843643904 2575897 33554432000 4194304 4338100 140734746055488 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=10589496 CPUtime=938.89 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 106122 0 0 0 88232 5657 0 0 25 0 5 0 522463542 10843643904 2575898 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3757.13
Current children cumulated vsize (KiB) 10589496

[startup+1002.3 s]
/proc/loadavg: 8.07 8.03 8.00 9/184 26821
/proc/meminfo: memFree=9566172/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=11000448 CPUtime=3996.61 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 2664467 0 0 0 386972 12689 0 0 25 0 5 0 522463394 11264458752 2658472 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 2750112 2658472 176 36 0 2748017 0
[pid=26789/tid=26791] ppid=26787 vsize=11000448 CPUtime=998.46 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 1542194 0 0 0 99335 511 0 0 25 0 5 0 522463542 11264458752 2658472 33554432000 4194304 4338100 140734746055488 18446744073709551615 4241252 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=11000448 CPUtime=997.62 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 140617 0 0 0 93756 6006 0 0 25 0 5 0 522463542 11264458752 2658472 33554432000 4194304 4338100 140734746055488 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=11000448 CPUtime=1000.19 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 826328 0 0 0 99866 153 0 0 25 0 5 0 522463542 11264458752 2658472 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664203710 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=11000448 CPUtime=998.81 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 115496 0 0 0 93869 6012 0 0 25 0 5 0 522463542 11264458752 2658472 33554432000 4194304 4338100 140734746055488 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3996.61
Current children cumulated vsize (KiB) 11000448

[startup+1062.3 s]
/proc/loadavg: 5.95 7.48 7.81 5/174 26840
/proc/meminfo: memFree=19974948/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=11326944 CPUtime=4236.08 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 2720483 0 0 0 410028 13580 0 0 25 0 5 0 522463394 11598790656 2714488 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 2831736 2714488 176 36 0 2829641 0
[pid=26789/tid=26791] ppid=26787 vsize=11326944 CPUtime=1058.4 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 1582601 0 0 0 105306 534 0 0 25 0 5 0 522463542 11598790656 2714488 33554432000 4194304 4338100 140734746055488 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=11326944 CPUtime=1057.37 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 149365 0 0 0 99296 6441 0 0 25 0 5 0 522463542 11598790656 2714488 33554432000 4194304 4338100 140734746055488 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=11326944 CPUtime=1060.16 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 826328 0 0 0 105863 153 0 0 25 0 5 0 522463542 11598790656 2714488 33554432000 4194304 4338100 140734746055488 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=11326944 CPUtime=1058.64 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 122357 0 0 0 99418 6446 0 0 25 0 5 0 522463542 11598790656 2714488 33554432000 4194304 4338100 140734746055488 18446744073709551615 4235674 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4236.08
Current children cumulated vsize (KiB) 11326944

[startup+1122.3 s]
/proc/loadavg: 4.76 6.86 7.58 5/174 26842
/proc/meminfo: memFree=19452760/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=11836808 CPUtime=4475.64 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 2850730 0 0 0 432989 14575 0 0 25 0 5 0 522463394 12120891392 2844735 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 2959202 2844735 176 36 0 2957107 0
[pid=26789/tid=26791] ppid=26787 vsize=11836808 CPUtime=1118.36 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 1676041 0 0 0 111268 568 0 0 25 0 5 0 522463542 12120891392 2844735 33554432000 4194304 4338100 140734746055488 18446744073709551615 4239479 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=11836808 CPUtime=1117.3 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 151039 0 0 0 104809 6921 0 0 25 0 5 0 522463542 12120891392 2844735 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=11836808 CPUtime=1120.01 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 860571 0 0 0 111842 159 0 0 25 0 5 0 522463542 12120891392 2844735 33554432000 4194304 4338100 140734746055488 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=11836808 CPUtime=1118.46 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 123247 0 0 0 104925 6921 0 0 25 0 5 0 522463542 12120891392 2844735 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4475.64
Current children cumulated vsize (KiB) 11836808

[startup+1182.3 s]
/proc/loadavg: 4.31 6.35 7.36 5/174 26843
/proc/meminfo: memFree=19028408/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=12243448 CPUtime=4715.21 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) S 26787 26789 26305 0 -1 4202496 2957242 0 0 0 455990 15531 0 0 25 0 5 0 522463394 12537290752 2951247 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 3060862 2951247 176 36 0 3058767 0
[pid=26789/tid=26791] ppid=26787 vsize=12243448 CPUtime=1178.14 cores=1,3,5,7
/proc/26789/task/26791/stat : 26791 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 1757111 0 0 0 117216 598 0 0 25 0 5 0 522463542 12537290752 2951247 33554432000 4194304 4338100 140734746055488 18446744073709551615 274664204155 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26789/tid=26792] ppid=26787 vsize=12243448 CPUtime=1177.27 cores=1,3,5,7
/proc/26789/task/26792/stat : 26792 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 153912 0 0 0 110344 7383 0 0 25 0 5 0 522463542 12537290752 2951247 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26789/tid=26793] ppid=26787 vsize=12243448 CPUtime=1179.87 cores=1,3,5,7
/proc/26789/task/26793/stat : 26793 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 880897 0 0 0 117824 163 0 0 25 0 5 0 522463542 12537290752 2951247 33554432000 4194304 4338100 140734746055488 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=12243448 CPUtime=1178.43 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4202560 125490 0 0 0 110461 7382 0 0 25 0 5 0 522463542 12537290752 2951247 33554432000 4194304 4338100 140734746055488 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4715.21
Current children cumulated vsize (KiB) 12243448



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.6 s]
/proc/loadavg: 4.22 6.20 7.28 2/171 26844
/proc/meminfo: memFree=19781420/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=0 CPUtime=4800.06 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) Z 26787 26789 26305 0 -1 4203548 2994409 0 0 0 464146 15860 0 0 18 0 2 0 522463394 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 0 0 0 0 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=0 CPUtime=1199.72 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4203588 125490 0 0 0 112431 7541 0 0 25 0 2 0 522463542 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.06
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.8 s]
/proc/loadavg: 4.22 6.20 7.28 2/171 26844
/proc/meminfo: memFree=19781420/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=0 CPUtime=4800.25 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) Z 26787 26789 26305 0 -1 4203548 2994409 0 0 0 464146 15879 0 0 18 0 2 0 522463394 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 0 0 0 0 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=0 CPUtime=1199.91 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4203588 125490 0 0 0 112431 7560 0 0 25 0 2 0 522463542 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

[startup+1203.9 s]
/proc/loadavg: 4.22 6.20 7.28 2/171 26844
/proc/meminfo: memFree=19781420/32951132 swapFree=67111356/67111528
[pid=26789] ppid=26787 vsize=0 CPUtime=4800.35 cores=1,3,5,7
/proc/26789/stat : 26789 (ttsth-5-4) Z 26787 26789 26305 0 -1 4203548 2994409 0 0 0 464146 15889 0 0 18 0 2 0 522463394 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/26789/statm: 0 0 0 0 0 0 0
[pid=26789/tid=26794] ppid=26787 vsize=0 CPUtime=1200.01 cores=1,3,5,7
/proc/26789/task/26794/stat : 26794 (ttsth-5-4) R 26787 26789 26305 0 -1 4203588 125490 0 0 0 112431 7570 0 0 25 0 2 0 522463542 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.35
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.97
CPU time (s): 4800.43
CPU user time (s): 4641.46
CPU system time (s): 158.969
CPU usage (%): 398.717
Max. virtual memory (cumulated for all children) (KiB): 12393592

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4641.46
system time used= 158.969
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2994409
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 13285
involuntary context switches= 33354

runsolver used 1.88171 second user time and 4.53531 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-29 22:23:58
IDJOB=3251704
IDBENCH=88229
IDSOLVER=1654
FILE ID=node131/3251704-1304108637
RUNJOBID= node131-1304107233-26622
PBS_JOBID= 13170641
Free space on /tmp= 71380 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m42-q585-n40-p30-s19690873.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251704-1304108637/watcher-3251704-1304108637 -o /tmp/evaluation-result-3251704-1304108637/solver-3251704-1304108637 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251704-1304108637.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e096925b96a6999132833f44e4f216d7
RANDOM SEED=1113430399

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      27396748 kB
Buffers:        205764 kB
Cached:        1521344 kB
SwapCached:        168 kB
Active:        3602128 kB
Inactive:      1639692 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      27396748 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            3148 kB
Writeback:           0 kB
AnonPages:     3514212 kB
Mapped:          15548 kB
Slab:           242692 kB
PageTables:      10980 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:  3728108 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71384 MiB
End job on node131 at 2011-04-29 22:44:03