Trace number 3262354

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.31 1202.26

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/
VanDerWaerden_pd_2-3-22_443.cnf
MD5SUM91ac0151536835225f21cbe5a3574a2d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.435933
Satisfiable
(Un)Satisfiability was proved
Number of variables222
Number of clauses26201
Sum of the clauses size115467
Maximum clause length22
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2294
Number of clauses of size 323935
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 51972

Solver Data 

0.01/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262354-1304092115.cnf 
0.01/0.03	c Using SATTIMEOUT of 4800
0.01/0.03	c Using SATRAM of 15500
0.01/0.03	c Size of subset node = 32
0.01/0.03	c Variables = 222
0.01/0.03	c Clauses = 26201
0.01/0.03	c Literals = 115467
0.01/0.03	c Reduced size:
0.01/0.03	c Variables = 222
0.01/0.03	c Clauses = 26201
0.01/0.03	c Literals = 115467
0.01/0.03	c a2
0.09/0.16	c a1
0.29/0.38	c Independent sub-problems = 1
0.29/0.38	c Sub-problem with 222 variables and 26201 clauses
0.29/0.38	c using permutation method 4
0.29/0.38	c count = 1
0.29/0.38	c Ordering finished 0.370 seconds
0.29/0.38	c mapped 0.380 seconds
0.29/0.38	c permuted 0.380 seconds
0.29/0.38	c about to sort
0.29/0.39	c sorted
0.39/0.47	c clauses now 26201
0.39/0.47	c init finished 0.390 seconds
0.39/0.47	c tree built 0.460 seconds
0.39/0.47	c 
0.39/0.47	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.47	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.47	c ................................ | ..........                    |                                  0.5         0.0             
0.39/0.47	c ................................ | ..........                  1 |                                  0.5         2.2             
0.39/0.47	c ................................ | ..........                  2 |                                  0.5         4.3             
0.39/0.47	c ................................ | ..........                  3 |                                  0.5         6.5             
0.39/0.47	c ................................ | ..........                  4 |                                  0.5         8.7             
0.39/0.47	c ................................ | ..........                  5 |                                  0.5        10.9             
0.39/0.49	c ................................ | ..........               1000 | 10           266                 0.5      1923.1        23501
96.02/24.44	c ................................ | ..........             403000 | 48067        125664             96.2      4189.2     21614941
192.29/48.54	c ................................ | ..........             611000 | 73090        192660            192.4      3175.0     33680488
288.17/72.57	c ................................ | ..........             783000 | 96234        257169            288.5      2714.4     44377683
384.03/96.50	c ................................ | ..........             943000 | 115584       296671            384.0      2455.5     54027869
479.90/120.57	c ................................ | *.........            1092000 | 134995       333218            480.2      2274.1     62370693
576.58/144.76	c ................................ | *.........            1232000 | 152391       373510            576.8      2135.8     70772556
672.44/168.74	c ................................ | *.........            1339000 | 166417       408991            672.6      1990.8     78000905
768.31/192.75	c ................................ | *.........            1438000 | 182711       434859            768.5      1871.1     84107399
863.81/216.69	c ................................ | *.........            1533000 | 195604       462002            864.2      1774.0     89582580
960.48/240.82	c ................................ | **........            1613000 | 207209       482808            960.5      1679.3     94869648
1056.32/264.81	c ................................ | **........            1695000 | 218336       506552           1056.4      1604.5    100078341
1152.60/288.96	c ................................ | **........            1772000 | 228736       527435           1152.8      1537.1    105469049
1248.07/312.88	c ................................ | **........            1847000 | 239975       548957           1248.4      1479.5    111310514
1343.94/336.85	c ................................ | **........            1929000 | 251593       568060           1344.2      1435.1    117232262
1440.60/361.02	c ................................ | ***.......            2005000 | 260548       591547           1440.7      1391.7    120960930
1536.88/385.13	c ................................ | ***.......            2086000 | 270018       614118           1537.0      1357.2    125929955
1632.35/409.03	c ................................ | ***.......            2153000 | 278595       633601           1632.5      1318.8    130371201
1728.61/433.16	c ................................ | ***.......            2223000 | 286953       654142           1728.9      1285.8    135028804
1824.48/457.18	c ................................ | ***.......            2286000 | 294477       672173           1824.8      1252.7    139251546
1920.36/481.17	c ................................ | ****......            2352000 | 301031       688431           1920.7      1224.6    142473628
2016.62/505.20	c ................................ | ****......            2416000 | 308071       705047           2016.7      1198.0    145880102
2113.30/529.43	c ................................ | ****......            2486000 | 315581       723578           2113.4      1176.3    149012261
2208.79/553.34	c ................................ | ****......            2553000 | 323688       741122           2208.9      1155.8    149981501
2304.23/577.23	c ................................ | ****......            2616000 | 330670       757350           2304.4      1135.2    152770546
2400.51/601.34	c ................................ | *****.....            2673000 | 337876       773325           2400.7      1113.4    155942278
2496.78/625.48	c ................................ | *****.....            2738000 | 345186       788786           2497.1      1096.5    158830769
2592.24/649.35	c ................................ | *****.....            2799000 | 351594       805312           2592.4      1079.7    160535047
2688.51/673.44	c ................................ | *****.....            2854000 | 357920       820494           2688.7      1061.5    162707223
2785.18/697.66	c ................................ | *****.....            2909000 | 365110       835341           2785.4      1044.4    165810090
2880.68/721.51	c ................................ | ******....            2966000 | 372253       849710           2880.7      1029.6    169471394
2975.73/745.36	c ................................ | ******....            3021000 | 379367       862601           2976.0      1015.1    173495550
3072.78/769.60	c ................................ | ******....            3086000 | 388043       876498           3072.8      1004.3    175278068
3169.05/793.78	c ................................ | ******....            3146000 | 395563       891053           3169.4       992.6    175278068
3265.33/817.80	c ................................ | ******....            3211000 | 403520       905666           3265.3       983.4    175982623
3361.19/841.82	c ................................ | *******...            3273000 | 410907       918712           3361.3       973.7    178740216
3456.68/865.71	c ................................ | *******...            3332000 | 417240       932990           3456.7       963.9    181738061
3553.36/889.95	c ................................ | *******...            3392000 | 424008       946266           3553.5       954.5    184533391
3649.62/914.03	c ................................ | *******...            3441000 | 430841       958719           3649.7       942.8    186916190
3745.08/937.98	c ................................ | *******...            3492000 | 436655       972445           3745.4       932.3    189475443
3840.94/961.95	c ................................ | ********..            3541000 | 442650       986269           3841.1       921.9    191655750
3936.80/985.90	c ................................ | ********..            3595000 | 448929       1003656          3936.8       913.2    191655750
4032.29/1009.88	c ................................ | ********..            3653000 | 455560       1015386          4032.6       905.9    191655750
4128.15/1033.88	c ................................ | ********..            3701000 | 461388       1027819          4128.5       896.5    192012799
4224.82/1058.09	c ................................ | ********..            3749000 | 468344       1038715          4225.2       887.3    193810400
4321.48/1082.23	c ................................ | *********.            3796000 | 474975       1053583          4321.6       878.4    195480776
4417.36/1106.23	c ................................ | *********.            3839000 | 479660       1066141          4417.5       869.0    197036223
4513.62/1130.34	c ................................ | *********.            3886000 | 485411       1078305          4513.8       860.9    199586624
4609.48/1154.35	c ................................ | *********.            3933000 | 491149       1090705          4609.7       853.2    201634977
4704.94/1178.23	c ................................ | *********.            3983000 | 497945       1103672          4705.1       846.5    203144446

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262354-1304092115/watcher-3262354-1304092115 -o /tmp/evaluation-result-3262354-1304092115/solver-3262354-1304092115 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262354-1304092115.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.68 4.92 6.53 2/177 21198
/proc/meminfo: memFree=30885648/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=9540 CPUtime=0 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) R 21196 21198 20765 0 -1 4202496 441 0 0 0 0 0 0 0 22 0 1 0 599442068 9768960 384 33554432000 4194304 4338100 140736162067232 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21198/statm: 2385 385 146 36 0 290 0

[startup+0.015731 s]
/proc/loadavg: 3.68 4.92 6.53 2/177 21198
/proc/meminfo: memFree=30885648/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=11160 CPUtime=0.01 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) R 21196 21198 20765 0 -1 4202496 836 0 0 0 1 0 0 0 22 0 1 0 599442068 11427840 779 33554432000 4194304 4338100 140736162067232 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21198/statm: 2790 780 147 36 0 695 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 11160

[startup+0.100729 s]
/proc/loadavg: 3.68 4.92 6.53 2/177 21198
/proc/meminfo: memFree=30885648/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=26316 CPUtime=0.09 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) R 21196 21198 20765 0 -1 4202496 3929 0 0 0 9 0 0 0 22 0 1 0 599442068 26947584 3740 33554432000 4194304 4338100 140736162067232 18446744073709551615 260019791558 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21198/statm: 6579 3740 353 36 0 4484 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26316

[startup+0.300677 s]
/proc/loadavg: 3.68 4.92 6.53 2/177 21198
/proc/meminfo: memFree=30885648/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=15136 CPUtime=0.29 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) R 21196 21198 20765 0 -1 4202496 4436 0 0 0 29 0 0 0 23 0 1 0 599442068 15499264 1819 33554432000 4194304 4338100 140736162067232 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21198/statm: 3784 1819 155 36 0 1689 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 15136

[startup+0.700576 s]
/proc/loadavg: 3.68 4.92 6.53 2/177 21198
/proc/meminfo: memFree=30885648/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=161312 CPUtime=1.36 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 13116 0 0 0 135 1 0 0 25 0 5 0 599442068 165183488 10382 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 40328 10383 176 36 0 38233 0
Current children cumulated CPU time (s) 1.36
Current children cumulated vsize (KiB) 161312

[startup+1.50043 s]
/proc/loadavg: 4.03 4.97 6.53 10/186 21207
/proc/meminfo: memFree=30759104/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=202372 CPUtime=4.55 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 23646 0 0 0 453 2 0 0 25 0 5 0 599442068 207228928 20912 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 50593 20913 176 36 0 48498 0
[pid=21198/tid=21204] ppid=21196 vsize=202372 CPUtime=1.02 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 3870 0 0 0 102 0 0 0 19 0 5 0 599442115 207228928 20913 33554432000 4194304 4338100 140736162067232 18446744073709551615 4285383 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=202372 CPUtime=1.01 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 3507 0 0 0 101 0 0 0 25 0 5 0 599442115 207228928 20914 33554432000 4194304 4338100 140736162067232 18446744073709551615 4265608 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=202372 CPUtime=1.01 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 3105 0 0 0 101 0 0 0 19 0 5 0 599442115 207228928 20914 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=202372 CPUtime=1.01 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 3819 0 0 0 101 0 0 0 25 0 5 0 599442115 207228928 20915 33554432000 4194304 4338100 140736162067232 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4.55
Current children cumulated vsize (KiB) 202372

[startup+3.10114 s]
/proc/loadavg: 4.03 4.97 6.53 10/186 21207
/proc/meminfo: memFree=30640060/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=272092 CPUtime=10.95 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 41497 0 0 0 1089 6 0 0 25 0 5 0 599442068 278622208 38763 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 68023 38764 176 36 0 65928 0
[pid=21198/tid=21204] ppid=21196 vsize=272092 CPUtime=2.62 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 8245 0 0 0 261 1 0 0 23 0 5 0 599442115 278622208 38764 33554432000 4194304 4338100 140736162067232 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=272092 CPUtime=2.61 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 8755 0 0 0 260 1 0 0 25 0 5 0 599442115 278622208 38765 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=272092 CPUtime=2.61 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 6402 0 0 0 260 1 0 0 23 0 5 0 599442115 278622208 38765 33554432000 4194304 4338100 140736162067232 18446744073709551615 4265557 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=272092 CPUtime=2.61 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 8748 0 0 0 260 1 0 0 25 0 5 0 599442115 278622208 38765 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.95
Current children cumulated vsize (KiB) 272092

[startup+6.30055 s]
/proc/loadavg: 4.03 4.97 6.53 10/186 21207
/proc/meminfo: memFree=30360068/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=391616 CPUtime=23.72 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 71343 0 0 0 2361 11 0 0 25 0 5 0 599442068 401014784 68609 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 97904 68610 176 36 0 95809 0
[pid=21198/tid=21204] ppid=21196 vsize=391616 CPUtime=5.82 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 17139 0 0 0 579 3 0 0 25 0 5 0 599442115 401014784 68610 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=391616 CPUtime=5.81 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 17348 0 0 0 578 3 0 0 25 0 5 0 599442115 401014784 68611 33554432000 4194304 4338100 140736162067232 18446744073709551615 4286296 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=391616 CPUtime=5.78 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 11783 0 0 0 577 1 0 0 25 0 5 0 599442115 401014784 68611 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=391616 CPUtime=5.81 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 15727 0 0 0 579 2 0 0 25 0 5 0 599442115 401014784 68611 33554432000 4194304 4338100 140736162067232 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.72
Current children cumulated vsize (KiB) 391616

[startup+12.7004 s]
/proc/loadavg: 4.64 5.07 6.55 9/186 21207
/proc/meminfo: memFree=29978788/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=573964 CPUtime=49.29 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 116182 0 0 0 4908 21 0 0 25 0 5 0 599442068 587739136 113448 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 143491 113448 176 36 0 141396 0
[pid=21198/tid=21204] ppid=21196 vsize=573964 CPUtime=12.22 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 31192 0 0 0 1216 6 0 0 25 0 5 0 599442115 587739136 113448 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=573964 CPUtime=12.21 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 28675 0 0 0 1217 4 0 0 25 0 5 0 599442115 587739136 113448 33554432000 4194304 4338100 140736162067232 18446744073709551615 4239509 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=573964 CPUtime=12.17 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 20528 0 0 0 1214 3 0 0 25 0 5 0 599442115 587739136 113448 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=573964 CPUtime=12.18 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 26439 0 0 0 1213 5 0 0 25 0 5 0 599442115 587739136 113448 33554432000 4194304 4338100 140736162067232 18446744073709551615 4256318 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.29
Current children cumulated vsize (KiB) 573964

[startup+25.501 s]
/proc/loadavg: 5.16 5.17 6.56 10/186 21207
/proc/meminfo: memFree=29395740/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=918552 CPUtime=100.41 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 185644 0 0 0 10005 36 0 0 25 0 5 0 599442068 940597248 182910 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 229638 182911 176 36 0 227543 0
[pid=21198/tid=21204] ppid=21196 vsize=918552 CPUtime=25.02 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 48585 0 0 0 2491 11 0 0 25 0 5 0 599442115 940597248 182912 33554432000 4194304 4338100 140736162067232 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=918552 CPUtime=25.01 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 47649 0 0 0 2493 8 0 0 25 0 5 0 599442115 940597248 182912 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=918552 CPUtime=24.9 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 33506 0 0 0 2483 7 0 0 25 0 5 0 599442115 940597248 182912 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=918552 CPUtime=24.98 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 46558 0 0 0 2490 8 0 0 25 0 5 0 599442115 940597248 182912 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.41
Current children cumulated vsize (KiB) 918552

[startup+51.1003 s]
/proc/loadavg: 6.19 5.41 6.61 10/186 21208
/proc/meminfo: memFree=28557052/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=1312212 CPUtime=202.67 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 284806 0 0 0 20210 57 0 0 25 0 5 0 599442068 1343705088 282072 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 328053 282072 176 36 0 325958 0
[pid=21198/tid=21204] ppid=21196 vsize=1312212 CPUtime=50.62 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 74685 0 0 0 5046 16 0 0 25 0 5 0 599442115 1343705088 282072 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=1312212 CPUtime=50.61 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 77258 0 0 0 5048 13 0 0 25 0 5 0 599442115 1343705088 282072 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=1312212 CPUtime=50.36 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 53982 0 0 0 5024 12 0 0 25 0 5 0 599442115 1343705088 282072 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=1312212 CPUtime=50.58 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 69533 0 0 0 5044 14 0 0 25 0 5 0 599442115 1343705088 282072 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.67
Current children cumulated vsize (KiB) 1312212

[startup+102.307 s]
/proc/loadavg: 7.34 5.86 6.69 9/186 21209
/proc/meminfo: memFree=27118492/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=1983604 CPUtime=407.22 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 452426 0 0 0 40628 94 0 0 25 0 5 0 599442068 2031210496 449692 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 495901 449692 176 36 0 493806 0
[pid=21198/tid=21204] ppid=21196 vsize=1983604 CPUtime=101.83 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 126068 0 0 0 10155 28 0 0 25 0 5 0 599442115 2031210496 449692 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032208590 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=1983604 CPUtime=101.73 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 122512 0 0 0 10150 23 0 0 25 0 5 0 599442115 2031210496 449692 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=1983604 CPUtime=101.37 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 82431 0 0 0 10118 19 0 0 25 0 5 0 599442115 2031210496 449692 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=1983604 CPUtime=101.78 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 112067 0 0 0 10157 21 0 0 25 0 5 0 599442115 2031210496 449692 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.22
Current children cumulated vsize (KiB) 1983604

[startup+162.301 s]
/proc/loadavg: 7.82 6.26 6.77 10/186 21211
/proc/meminfo: memFree=25872108/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=2617932 CPUtime=646.88 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 611356 0 0 0 64562 126 0 0 25 0 5 0 599442068 2680762368 608622 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 654483 608622 176 36 0 652388 0
[pid=21198/tid=21204] ppid=21196 vsize=2617932 CPUtime=161.82 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 176168 0 0 0 16146 36 0 0 25 0 5 0 599442115 2680762368 608622 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=2617932 CPUtime=161.41 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 173056 0 0 0 16107 34 0 0 25 0 5 0 599442115 2680762368 608622 33554432000 4194304 4338100 140736162067232 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=2617932 CPUtime=161.36 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 106124 0 0 0 16112 24 0 0 25 0 5 0 599442115 2680762368 608622 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=2617932 CPUtime=161.78 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 146660 0 0 0 16150 28 0 0 25 0 5 0 599442115 2680762368 608622 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.88
Current children cumulated vsize (KiB) 2617932


################
# More data... #
################

/proc/meminfo: memFree=19135516/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=5842744 CPUtime=3043.64 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1413698 0 0 0 304055 309 0 0 25 0 5 0 599442068 5982969856 1410964 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1460686 1410964 176 36 0 1458591 0
[pid=21198/tid=21204] ppid=21196 vsize=5842744 CPUtime=761.84 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 396384 0 0 0 76096 88 0 0 25 0 5 0 599442115 5982969856 1410964 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=5842744 CPUtime=761.28 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 355054 0 0 0 76052 76 0 0 25 0 5 0 599442115 5982969856 1410964 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=5842744 CPUtime=758.25 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 260503 0 0 0 75766 59 0 0 25 0 5 0 599442115 5982969856 1410964 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=5842744 CPUtime=761.77 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 392409 0 0 0 76093 84 0 0 25 0 5 0 599442115 5982969856 1410964 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3043.64
Current children cumulated vsize (KiB) 5842744

[startup+822.301 s]
/proc/loadavg: 8.13 7.89 7.43 10/186 21237
/proc/meminfo: memFree=18952580/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=5880808 CPUtime=3283.3 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1423805 0 0 0 328018 312 0 0 25 0 5 0 599442068 6021947392 1421071 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1470202 1421071 176 36 0 1468107 0
[pid=21198/tid=21204] ppid=21196 vsize=5880808 CPUtime=821.84 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 402076 0 0 0 82094 90 0 0 25 0 5 0 599442115 6021947392 1421071 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=5880808 CPUtime=821.27 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 359406 0 0 0 82050 77 0 0 25 0 5 0 599442115 6021947392 1421071 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=5880808 CPUtime=817.93 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 260530 0 0 0 81734 59 0 0 25 0 5 0 599442115 6021947392 1421071 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=5880808 CPUtime=821.76 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 392445 0 0 0 82092 84 0 0 25 0 5 0 599442115 6021947392 1421071 33554432000 4194304 4338100 140736162067232 18446744073709551615 4256400 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3283.3
Current children cumulated vsize (KiB) 5880808

[startup+882.301 s]
/proc/loadavg: 8.12 7.92 7.47 9/186 21239
/proc/meminfo: memFree=18546564/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=6103160 CPUtime=3522.98 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1479514 0 0 0 351972 326 0 0 25 0 5 0 599442068 6249635840 1476780 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1525790 1476780 176 36 0 1523695 0
[pid=21198/tid=21204] ppid=21196 vsize=6103160 CPUtime=881.84 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 427075 0 0 0 88089 95 0 0 25 0 5 0 599442115 6249635840 1476780 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=6103160 CPUtime=881.27 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 373676 0 0 0 88046 81 0 0 25 0 5 0 599442115 6249635840 1476780 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=6103160 CPUtime=877.6 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 260550 0 0 0 87701 59 0 0 25 0 5 0 599442115 6249635840 1476780 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=6103160 CPUtime=881.76 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 408865 0 0 0 88088 88 0 0 25 0 5 0 599442115 6249635840 1476780 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254347 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3522.98
Current children cumulated vsize (KiB) 6103160

[startup+942.301 s]
/proc/loadavg: 8.11 7.95 7.50 10/186 21240
/proc/meminfo: memFree=18150208/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=6300620 CPUtime=3762.66 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1528466 0 0 0 375926 340 0 0 25 0 5 0 599442068 6451834880 1525732 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1575155 1525732 176 36 0 1573060 0
[pid=21198/tid=21204] ppid=21196 vsize=6300620 CPUtime=941.83 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 447582 0 0 0 94082 101 0 0 25 0 5 0 599442115 6451834880 1525732 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=6300620 CPUtime=941.28 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 387219 0 0 0 94043 85 0 0 25 0 5 0 599442115 6451834880 1525732 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=6300620 CPUtime=937.27 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 260586 0 0 0 93668 59 0 0 25 0 5 0 599442115 6451834880 1525732 33554432000 4194304 4338100 140736162067232 18446744073709551615 4239509 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=6300620 CPUtime=941.76 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 423731 0 0 0 94084 92 0 0 25 0 5 0 599442115 6451834880 1525732 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3762.66
Current children cumulated vsize (KiB) 6300620

[startup+1002.3 s]
/proc/loadavg: 8.10 7.97 7.54 10/186 21242
/proc/meminfo: memFree=17808288/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=6418628 CPUtime=4002.31 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1548427 0 0 0 399884 347 0 0 25 0 5 0 599442068 6572675072 1545693 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1604657 1545693 176 36 0 1602562 0
[pid=21198/tid=21204] ppid=21196 vsize=6418628 CPUtime=1001.84 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 459048 0 0 0 100079 105 0 0 25 0 5 0 599442115 6572675072 1545693 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=6418628 CPUtime=1001.04 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 391295 0 0 0 100018 86 0 0 25 0 5 0 599442115 6572675072 1545693 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=6418628 CPUtime=997.16 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 261866 0 0 0 99656 60 0 0 25 0 5 0 599442115 6572675072 1545693 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=6418628 CPUtime=1001.77 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 426870 0 0 0 100083 94 0 0 25 0 5 0 599442115 6572675072 1545693 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4002.31
Current children cumulated vsize (KiB) 6418628

[startup+1062.3 s]
/proc/loadavg: 8.15 8.00 7.57 10/186 21244
/proc/meminfo: memFree=17419492/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=6493964 CPUtime=4242 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1568056 0 0 0 423847 353 0 0 25 0 5 0 599442068 6649819136 1565322 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1623491 1565322 176 36 0 1621396 0
[pid=21198/tid=21204] ppid=21196 vsize=6493964 CPUtime=1061.83 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 459094 0 0 0 106078 105 0 0 25 0 5 0 599442115 6649819136 1565322 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=6493964 CPUtime=1060.73 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 400937 0 0 0 105984 89 0 0 25 0 5 0 599442115 6649819136 1565322 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=6493964 CPUtime=1057.16 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 261905 0 0 0 105656 60 0 0 25 0 5 0 599442115 6649819136 1565322 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=6493964 CPUtime=1061.76 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 436772 0 0 0 106080 96 0 0 25 0 5 0 599442115 6649819136 1565322 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4242
Current children cumulated vsize (KiB) 6493964

[startup+1122.3 s]
/proc/loadavg: 8.11 8.01 7.60 10/186 21245
/proc/meminfo: memFree=17016672/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=6638248 CPUtime=4481.67 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1604068 0 0 0 447806 361 0 0 25 0 5 0 599442068 6797565952 1601334 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1659562 1601334 176 36 0 1657467 0
[pid=21198/tid=21204] ppid=21196 vsize=6638248 CPUtime=1121.84 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 463091 0 0 0 112077 107 0 0 25 0 5 0 599442115 6797565952 1601334 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=6638248 CPUtime=1120.4 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 414498 0 0 0 111948 92 0 0 25 0 5 0 599442115 6797565952 1601334 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=6638248 CPUtime=1117.17 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 261949 0 0 0 111657 60 0 0 25 0 5 0 599442115 6797565952 1601334 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=6638248 CPUtime=1121.76 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 455182 0 0 0 112077 99 0 0 25 0 5 0 599442115 6797565952 1601334 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4481.67
Current children cumulated vsize (KiB) 6638248

[startup+1182.3 s]
/proc/loadavg: 8.09 8.02 7.63 10/186 21247
/proc/meminfo: memFree=16627120/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=6782400 CPUtime=4721.33 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1640869 0 0 0 471762 371 0 0 25 0 5 0 599442068 6945177600 1638135 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1695600 1638135 176 36 0 1693505 0
[pid=21198/tid=21204] ppid=21196 vsize=6782400 CPUtime=1181.84 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 478270 0 0 0 118073 111 0 0 25 0 5 0 599442115 6945177600 1638135 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=6782400 CPUtime=1180.08 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 418151 0 0 0 117915 93 0 0 25 0 5 0 599442115 6945177600 1638135 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=6782400 CPUtime=1177.17 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 261986 0 0 0 117657 60 0 0 25 0 5 0 599442115 6945177600 1638135 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=6782400 CPUtime=1181.74 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 473114 0 0 0 118069 105 0 0 25 0 5 0 599442115 6945177600 1638135 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4721.33
Current children cumulated vsize (KiB) 6782400



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202 s]
/proc/loadavg: 8.06 8.02 7.64 6/184 21249
/proc/meminfo: memFree=16799528/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=6829320 CPUtime=4800.04 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) S 21196 21198 20765 0 -1 4202496 1652692 0 0 0 479630 374 0 0 25 0 5 0 599442068 6993223680 1649958 33554432000 4194304 4338100 140736162067232 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 1707330 1649958 176 36 0 1705235 0
[pid=21198/tid=21204] ppid=21196 vsize=6829320 CPUtime=1201.55 cores=1,3,5,7
/proc/21198/task/21204/stat : 21204 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 483804 0 0 0 120043 112 0 0 25 0 5 0 599442115 6993223680 1649958 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=6829320 CPUtime=1199.67 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 418165 0 0 0 119874 93 0 0 25 0 5 0 599442115 6993223680 1649958 33554432000 4194304 4338100 140736162067232 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21198/tid=21206] ppid=21196 vsize=6829320 CPUtime=1196.87 cores=1,3,5,7
/proc/21198/task/21206/stat : 21206 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 262004 0 0 0 119627 60 0 0 25 0 5 0 599442115 6993223680 1649958 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21198/tid=21207] ppid=21196 vsize=6829320 CPUtime=1201.44 cores=1,3,5,7
/proc/21198/task/21207/stat : 21207 (ttsth-5-4) R 21196 21198 20765 0 -1 4202560 479371 0 0 0 120038 106 0 0 25 0 5 0 599442115 6993223680 1649958 33554432000 4194304 4338100 140736162067232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 6829320

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.2 s]
/proc/loadavg: 8.06 8.02 7.64 3/182 21250
/proc/meminfo: memFree=26798568/32951124 swapFree=67111464/67111528
[pid=21198] ppid=21196 vsize=0 CPUtime=4800.25 cores=1,3,5,7
/proc/21198/stat : 21198 (ttsth-5-4) Z 21196 21198 20765 0 -1 4203548 1652692 0 0 0 479632 393 0 0 18 0 2 0 599442068 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/21198/statm: 0 0 0 0 0 0 0
[pid=21198/tid=21205] ppid=21196 vsize=0 CPUtime=1199.86 cores=1,3,5,7
/proc/21198/task/21205/stat : 21205 (ttsth-5-4) R 21196 21198 20765 0 -1 4203588 418165 0 0 0 119874 112 0 0 25 0 2 0 599442115 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.26
CPU time (s): 4800.31
CPU user time (s): 4796.33
CPU system time (s): 3.98039
CPU usage (%): 399.275
Max. virtual memory (cumulated for all children) (KiB): 6829320

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.33
system time used= 3.98039
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1652692
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 460
involuntary context switches= 33055

runsolver used 2.05669 second user time and 4.51631 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 17:48:35
IDJOB=3262354
IDBENCH=83293
IDSOLVER=1654
FILE ID=node110/3262354-1304092115
RUNJOBID= node110-1304092115-21179
PBS_JOBID= 13170568
Free space on /tmp= 72840 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/VanDerWaerden_pd_2-3-22_443.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262354-1304092115/watcher-3262354-1304092115 -o /tmp/evaluation-result-3262354-1304092115/solver-3262354-1304092115 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262354-1304092115.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 91ac0151536835225f21cbe5a3574a2d
RANDOM SEED=5293113

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30885928 kB
Buffers:        898904 kB
Cached:         692968 kB
SwapCached:         64 kB
Active:         121464 kB
Inactive:      1538516 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30885928 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:             840 kB
Writeback:           0 kB
AnonPages:       68276 kB
Mapped:          14876 kB
Slab:           341836 kB
PageTables:       4720 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   210120 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72840 MiB
End job on node110 at 2011-04-29 18:08:39