Trace number 3263850

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.26 1222.25

General information on the benchmark

NameSAT03/handmade/stanion/hwb/
hwb-n30-02-S77299857.shuffled-as.sat03-1633.cnf
MD5SUMbdf2c49c74bfdbd3bd4663dca402d24f
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables194
Number of clauses954
Sum of the clauses size3252
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 210
Number of clauses of size 3544
Number of clauses of size 4400
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263850-1304098484.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 194
0.00/0.00	c Clauses = 954
0.00/0.00	c Literals = 3252
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 194
0.00/0.00	c Clauses = 954
0.00/0.00	c Literals = 3252
0.00/0.00	c a2
0.00/0.00	c a1
0.19/0.25	c Independent sub-problems = 1
0.19/0.25	c Sub-problem with 194 variables and 954 clauses
0.19/0.25	c using permutation method 4
0.19/0.25	c count = 3
0.19/0.25	c Ordering finished 0.250 seconds
0.19/0.25	c mapped 0.250 seconds
0.19/0.25	c permuted 0.250 seconds
0.19/0.25	c about to sort
0.19/0.25	c sorted
0.19/0.25	c clauses now 954
0.19/0.25	c init finished 0.250 seconds
0.19/0.25	c tree built 0.250 seconds
0.19/0.25	c 
0.19/0.25	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.25	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.25	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.25	c ................................ | ..........                  1 |                                  0.2         4.0             
0.19/0.25	c ................................ | ..........                  2 |                                  0.2         8.0             
0.19/0.25	c ................................ | ..........                  3 |                                  0.2        12.0             
0.19/0.25	c ................................ | ..........                  4 |                                  0.2        16.0             
0.19/0.25	c ................................ | ..........                  5 |                                  0.2        20.0             
0.42/0.30	c ................................ | ..........               1000 | 21           522                 0.3      3846.2        26443
95.95/24.42	c ................................ | ..........            1214000 | 64221        1390538            96.0     12639.3     25593683
191.87/48.77	c ................................ | ..........            2205000 | 112052       2107203           192.2     11474.8     30120719
287.98/73.13	c ................................ | ..........            3104000 | 165042       2707469           288.1     10774.0     44005856
383.74/97.48	c ................................ | ..........            3822000 | 208357       3295166           384.1      9951.1     51758076
479.90/121.73	c ................................ | *.........            4479000 | 248486       3785838           480.0      9330.7     51758076
575.84/146.16	c ................................ | *.........            5208000 | 289077       4289008           576.1      9039.9     51758076
671.74/170.49	c ................................ | *.........            6020000 | 327298       4791605           672.1      8957.1     51758076
768.01/194.83	c ................................ | *.........            6763000 | 367014       5342093           768.2      8804.0     53300991
863.79/219.16	c ................................ | *.........            7459000 | 403678       5840511           864.0      8632.7     56166736
959.86/243.53	c ................................ | **........            8167000 | 439947       6328005           960.0      8507.3     56712225
1055.75/267.86	c ................................ | **........            8955000 | 478454       6841251          1056.0      8480.0     57309642
1152.01/292.21	c ................................ | **........            9660000 | 518032       7307002          1152.1      8385.0     59614865
1247.88/316.75	c ................................ | **........           10324000 | 555100       7770988          1248.1      8271.9     63768152
1343.94/341.25	c ................................ | **........           10921000 | 590922       8247962          1344.1      8125.0     65978703
1439.86/365.74	c ................................ | ***.......           11523000 | 624444       8707022          1440.0      8001.9     67872048
1535.82/390.28	c ................................ | ***.......           12164000 | 660587       9195113          1536.2      7918.5     67872048
1632.02/414.62	c ................................ | ***.......           12754000 | 694340       9618622          1632.1      7814.6     68223748
1727.87/439.03	c ................................ | ***.......           13362000 | 727451       10055678         1728.0      7732.5     68223748
1824.12/463.54	c ................................ | ***.......           13915000 | 762485       10496575         1824.2      7628.2     68223748
1919.82/487.87	c ................................ | ****......           14395000 | 793129       10885340         1920.1      7496.9     68223748
2016.03/512.31	c ................................ | ****......           14965000 | 826120       11324554         2016.1      7422.8     68223748
2111.63/536.59	c ................................ | ****......           15500000 | 857682       11715159         2112.0      7338.9     68748165
2207.96/561.02	c ................................ | ****......           16113000 | 895205       12135134         2208.0      7297.5     70693973
2303.81/585.55	c ................................ | ****......           16786000 | 929177       12544760         2304.0      7285.6     73028165
2399.99/610.05	c ................................ | *****.....           17352000 | 967337       12943085         2400.2      7229.4     76694178
2495.97/634.53	c ................................ | *****.....           17845000 | 995078       13316359         2496.1      7149.2     78058300
2592.00/658.96	c ................................ | *****.....           18263000 | 1024902      13698927         2592.2      7045.3     79753020
2688.76/683.50	c ................................ | *****.....           18740000 | 1054119      14078530         2688.8      6969.8     79889281
2785.76/708.21	c ................................ | *****.....           19340000 | 1084033      14487924         2785.8      6942.3     79889281
2879.75/732.19	c ................................ | ******....           19888000 | 1115204      14881582         2880.1      6905.2     79889281
2975.71/756.60	c ................................ | ******....           20409000 | 1145536      15256387         2976.1      6857.6     82155654
3072.18/781.11	c ................................ | ******....           20934000 | 1172812      15618290         3072.2      6813.9     85078923
3167.70/805.47	c ................................ | ******....           21428000 | 1203657      16011030         3168.0      6763.9     85417494
3263.89/829.99	c ................................ | ******....           22001000 | 1232647      16384842         3264.1      6740.3     87495927
3360.09/854.40	c ................................ | *******...           22579000 | 1264317      16785254         3360.1      6719.7     87495927
3455.65/878.79	c ................................ | *******...           23097000 | 1294834      17149625         3456.0      6683.1     88768289
3552.01/903.33	c ................................ | *******...           23716000 | 1329162      17536604         3552.1      6676.5     90082608
3647.79/927.85	c ................................ | *******...           24360000 | 1363257      17955256         3648.0      6677.6     93397900
3743.91/952.54	c ................................ | *******...           24889000 | 1397996      18359003         3744.1      6647.6     96303624
3839.83/977.35	c ................................ | ********..           25492000 | 1429688      18739433         3840.0      6638.5     98976315
3935.84/1001.94	c ................................ | ********..           26030000 | 1460729      19125702         3936.0      6613.3     98976315
4031.90/1026.33	c ................................ | ********..           26575000 | 1491366      19507615         4032.0      6591.0     98976315
4127.79/1050.89	c ................................ | ********..           27172000 | 1523038      19876315         4128.1      6582.3    102138692
4223.99/1075.72	c ................................ | ********..           27726000 | 1551142      20242536         4224.1      6563.8    102867442
4320.87/1100.38	c ................................ | *********.           28200000 | 1583047      20612043         4321.2      6525.9    102867442
4415.81/1124.57	c ................................ | *********.           28686000 | 1609881      20962871         4416.1      6495.7    103175172
4511.99/1149.03	c ................................ | *********.           29185000 | 1639403      21342802         4512.1      6468.1    103175172
4607.93/1173.32	c ................................ | *********.           29579000 | 1669042      21696094         4608.0      6419.0    103175172
4704.19/1197.71	c ................................ | *********.           30039000 | 1697879      22073997         4704.2      6385.5    103175172
4799.81/1222.07	c CPU limit (4800) reached
4799.81/1222.07	c Time = 4800.07 seconds
4799.81/1222.07	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263850-1304098484/watcher-3263850-1304098484 -o /tmp/evaluation-result-3263850-1304098484/solver-3263850-1304098484 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263850-1304098484.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.12 8.06 7.94 1/173 13035
/proc/meminfo: memFree=26624916/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=8972 CPUtime=0 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) R 13033 13035 12928 0 -1 4202496 316 0 0 0 0 0 0 0 25 0 1 0 521447088 9187328 259 33554432000 4194304 4338100 140733263103008 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/13035/statm: 2243 259 151 36 0 148 0

[startup+0.032486 s]
/proc/loadavg: 8.12 8.06 7.94 1/173 13035
/proc/meminfo: memFree=26624916/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=8972 CPUtime=0.03 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) R 13033 13035 12928 0 -1 4202496 317 0 0 0 3 0 0 0 25 0 1 0 521447088 9187328 260 33554432000 4194304 4338100 140733263103008 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/13035/statm: 2243 260 152 36 0 148 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 8972

[startup+0.100476 s]
/proc/loadavg: 8.12 8.06 7.94 1/173 13035
/proc/meminfo: memFree=26624916/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=8972 CPUtime=0.09 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) R 13033 13035 12928 0 -1 4202496 317 0 0 0 9 0 0 0 25 0 1 0 521447088 9187328 260 33554432000 4194304 4338100 140733263103008 18446744073709551615 4295667 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/13035/statm: 2243 260 152 36 0 148 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 8972

[startup+0.300406 s]
/proc/loadavg: 8.12 8.06 7.94 1/173 13035
/proc/meminfo: memFree=26624916/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=130504 CPUtime=0.42 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 2495 0 0 0 42 0 0 0 25 0 5 0 521447088 133636096 2438 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 32626 2439 173 36 0 30531 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 130504

[startup+0.700339 s]
/proc/loadavg: 8.12 8.06 7.94 1/173 13035
/proc/meminfo: memFree=26624916/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=234220 CPUtime=2 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 12304 0 0 0 199 1 0 0 25 0 5 0 521447088 239841280 12247 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 58555 12247 173 36 0 56460 0
Current children cumulated CPU time (s) 2
Current children cumulated vsize (KiB) 234220

[startup+1.5002 s]
/proc/loadavg: 7.87 8.01 7.92 6/180 13067
/proc/meminfo: memFree=26556020/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=293520 CPUtime=5.2 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 26819 0 0 0 516 4 0 0 25 0 5 0 521447088 300564480 26754 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 73380 26754 175 36 0 71285 0
[pid=13035/tid=13037] ppid=13033 vsize=293520 CPUtime=1.23 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 6927 0 0 0 122 1 0 0 19 0 5 0 521447114 300564480 26755 33554432000 4194304 4338100 140733263103008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=293520 CPUtime=1.22 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 2495 0 0 0 122 0 0 0 25 0 5 0 521447114 300564480 26756 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254010 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=293520 CPUtime=1.22 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 7416 0 0 0 122 0 0 0 25 0 5 0 521447114 300564480 26757 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=293520 CPUtime=1.23 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 9075 0 0 0 122 1 0 0 19 0 5 0 521447114 300564480 26757 33554432000 4194304 4338100 140733263103008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5.2
Current children cumulated vsize (KiB) 293520

[startup+3.10093 s]
/proc/loadavg: 7.87 8.01 7.92 9/184 13071
/proc/meminfo: memFree=26394068/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=401352 CPUtime=11.57 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 54916 0 0 0 1149 8 0 0 25 0 5 0 521447088 410984448 54851 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 100338 54851 175 36 0 98243 0
[pid=13035/tid=13037] ppid=13033 vsize=401352 CPUtime=2.83 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 15257 0 0 0 281 2 0 0 24 0 5 0 521447114 410984448 54851 33554432000 4194304 4338100 140733263103008 18446744073709551615 242969945838 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=401352 CPUtime=2.82 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 5278 0 0 0 281 1 0 0 18 0 5 0 521447114 410984448 54852 33554432000 4194304 4338100 140733263103008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=401352 CPUtime=2.81 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 15459 0 0 0 280 1 0 0 18 0 5 0 521447114 410984448 54852 33554432000 4194304 4338100 140733263103008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=401352 CPUtime=2.83 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 18014 0 0 0 280 3 0 0 24 0 5 0 521447114 410984448 54852 33554432000 4194304 4338100 140733263103008 18446744073709551615 4263456 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.57
Current children cumulated vsize (KiB) 401352

[startup+6.30037 s]
/proc/loadavg: 7.87 8.01 7.92 9/184 13071
/proc/meminfo: memFree=26064236/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=581956 CPUtime=24.31 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 100568 0 0 0 2414 17 0 0 25 0 5 0 521447088 595922944 100503 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 145489 100503 175 36 0 143394 0
[pid=13035/tid=13037] ppid=13033 vsize=581956 CPUtime=6.01 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 28656 0 0 0 596 5 0 0 25 0 5 0 521447114 595922944 100503 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982364419 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=581956 CPUtime=6.01 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 10510 0 0 0 599 2 0 0 25 0 5 0 521447114 595922944 100503 33554432000 4194304 4338100 140733263103008 18446744073709551615 4230411 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=581956 CPUtime=5.99 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 27708 0 0 0 596 3 0 0 25 0 5 0 521447114 595922944 100503 33554432000 4194304 4338100 140733263103008 18446744073709551615 4253933 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=581956 CPUtime=6.02 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 32786 0 0 0 597 5 0 0 18 0 5 0 521447114 595922944 100503 33554432000 4194304 4338100 140733263103008 18446744073709551615 242969945135 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.31
Current children cumulated vsize (KiB) 581956

[startup+12.7003 s]
/proc/loadavg: 7.97 8.02 7.93 9/184 13071
/proc/meminfo: memFree=25507144/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=853024 CPUtime=49.67 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 170080 0 0 0 4937 30 0 0 25 0 5 0 521447088 873496576 170016 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 213256 170016 175 36 0 211161 0
[pid=13035/tid=13037] ppid=13033 vsize=853024 CPUtime=12.33 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 46611 0 0 0 1225 8 0 0 25 0 5 0 521447114 873496576 170016 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=853024 CPUtime=12.36 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 23888 0 0 0 1232 4 0 0 25 0 5 0 521447114 873496576 170016 33554432000 4194304 4338100 140733263103008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=853024 CPUtime=12.33 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 45537 0 0 0 1226 7 0 0 25 0 5 0 521447114 873496576 170017 33554432000 4194304 4338100 140733263103008 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=853024 CPUtime=12.36 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 53137 0 0 0 1227 9 0 0 25 0 5 0 521447114 873496576 170017 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.67
Current children cumulated vsize (KiB) 853024

[startup+25.5011 s]
/proc/loadavg: 7.98 8.02 7.93 9/184 13072
/proc/meminfo: memFree=25053668/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=1089440 CPUtime=100.33 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 227829 0 0 0 9987 46 0 0 25 0 5 0 521447088 1115586560 227764 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 272360 227764 175 36 0 270265 0
[pid=13035/tid=13037] ppid=13033 vsize=1089440 CPUtime=24.97 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 66831 0 0 0 2483 14 0 0 25 0 5 0 521447114 1115586560 227764 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=1089440 CPUtime=25.08 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 38996 0 0 0 2500 8 0 0 25 0 5 0 521447114 1115586560 227764 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=1089440 CPUtime=24.98 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 56071 0 0 0 2489 9 0 0 25 0 5 0 521447114 1115586560 227764 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=1089440 CPUtime=24.99 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 65023 0 0 0 2487 12 0 0 25 0 5 0 521447114 1115586560 227764 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 100.33
Current children cumulated vsize (KiB) 1089440

[startup+51.1007 s]
/proc/loadavg: 8.06 8.03 7.93 9/184 13073
/proc/meminfo: memFree=24498760/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=1345616 CPUtime=201.36 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 287002 0 0 0 20069 67 0 0 25 0 5 0 521447088 1377910784 286937 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 336404 286937 175 36 0 334309 0
[pid=13035/tid=13037] ppid=13033 vsize=1345616 CPUtime=50.08 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 97801 0 0 0 4984 24 0 0 25 0 5 0 521447114 1377910784 286937 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=1345616 CPUtime=50.41 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 48532 0 0 0 5028 13 0 0 25 0 5 0 521447114 1377910784 286937 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=1345616 CPUtime=50.29 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 64474 0 0 0 5016 13 0 0 25 0 5 0 521447114 1377910784 286937 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=1345616 CPUtime=50.29 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 75287 0 0 0 5014 15 0 0 18 0 5 0 521447114 1377910784 286937 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 201.36
Current children cumulated vsize (KiB) 1345616

[startup+102.307 s]
/proc/loadavg: 8.03 8.03 7.93 9/184 13074
/proc/meminfo: memFree=23222672/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=2172300 CPUtime=403.2 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 484705 0 0 0 40201 119 0 0 25 0 5 0 521447088 2224435200 484640 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 543075 484640 175 36 0 540980 0
[pid=13035/tid=13037] ppid=13033 vsize=2172300 CPUtime=100.09 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 167900 0 0 0 9967 42 0 0 25 0 5 0 521447114 2224435200 484640 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=2172300 CPUtime=101.03 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 90059 0 0 0 10079 24 0 0 25 0 5 0 521447114 2224435200 484640 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=2172300 CPUtime=100.83 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 97110 0 0 0 10059 24 0 0 25 0 5 0 521447114 2224435200 484640 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=2172300 CPUtime=100.96 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 128728 0 0 0 10069 27 0 0 25 0 5 0 521447114 2224435200 484640 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 403.2
Current children cumulated vsize (KiB) 2172300

[startup+162.301 s]
/proc/loadavg: 7.89 7.99 7.92 9/184 13075
/proc/meminfo: memFree=22792740/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=2183580 CPUtime=639.74 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 501698 0 0 0 63833 141 0 0 25 0 5 0 521447088 2235985920 501633 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 545895 501633 175 36 0 543800 0
[pid=13035/tid=13037] ppid=13033 vsize=2183580 CPUtime=158.93 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 170374 0 0 0 15843 50 0 0 19 0 5 0 521447114 2235985920 501633 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=2183580 CPUtime=160.26 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 95313 0 0 0 15996 30 0 0 25 0 5 0 521447114 2235985920 501633 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=2183580 CPUtime=160.1 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 102214 0 0 0 15982 28 0 0 25 0 5 0 521447114 2235985920 501633 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=2183580 CPUtime=160.16 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 132889 0 0 0 15985 31 0 0 25 0 5 0 521447114 2235985920 501633 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 639.74
Current children cumulated vsize (KiB) 2183580


################
# More data... #
################

/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 123447 0 0 0 69213 74 0 0 25 0 5 0 521447114 3616391168 823858 33554432000 4194304 4338100 140733263103008 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=3531632 CPUtime=691.82 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 138709 0 0 0 69114 68 0 0 25 0 5 0 521447114 3616391168 823858 33554432000 4194304 4338100 140733263103008 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=3531632 CPUtime=691.54 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 151429 0 0 0 69088 66 0 0 25 0 5 0 521447114 3616391168 823858 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2762.73
Current children cumulated vsize (KiB) 3531632

[startup+762.306 s]
/proc/loadavg: 8.08 7.98 7.92 9/184 13091
/proc/meminfo: memFree=19278204/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=3774896 CPUtime=2998.52 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 859419 0 0 0 299454 398 0 0 25 0 5 0 521447088 3865493504 859354 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 943724 859354 175 36 0 941629 0
[pid=13035/tid=13037] ppid=13033 vsize=3774896 CPUtime=744.92 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 436764 0 0 0 74314 178 0 0 25 0 5 0 521447114 3865493504 859354 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=3774896 CPUtime=752.03 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 125215 0 0 0 75125 78 0 0 25 0 5 0 521447114 3865493504 859354 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=3774896 CPUtime=750.82 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 141383 0 0 0 75011 71 0 0 20 0 5 0 521447114 3865493504 859354 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=3774896 CPUtime=750.45 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 155149 0 0 0 74976 69 0 0 25 0 5 0 521447114 3865493504 859354 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2998.52
Current children cumulated vsize (KiB) 3774896

[startup+822.301 s]
/proc/loadavg: 7.84 7.94 7.91 9/184 13093
/proc/meminfo: memFree=18941536/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=3905724 CPUtime=3234.2 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 895967 0 0 0 322996 424 0 0 25 0 5 0 521447088 3999461376 895902 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 976431 895902 175 36 0 974336 0
[pid=13035/tid=13037] ppid=13033 vsize=3905724 CPUtime=803.49 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 468678 0 0 0 80156 193 0 0 25 0 5 0 521447114 3999461376 895902 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=3905724 CPUtime=811.13 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 127034 0 0 0 81031 82 0 0 25 0 5 0 521447114 3999461376 895902 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=3905724 CPUtime=809.99 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 142779 0 0 0 80924 75 0 0 25 0 5 0 521447114 3999461376 895902 33554432000 4194304 4338100 140733263103008 18446744073709551615 242969945534 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=3905724 CPUtime=809.31 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 156568 0 0 0 80858 73 0 0 25 0 5 0 521447114 3999461376 895902 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254413 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3234.2
Current children cumulated vsize (KiB) 3905724

[startup+882.3 s]
/proc/loadavg: 7.94 7.94 7.91 9/184 13095
/proc/meminfo: memFree=18624824/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=4131572 CPUtime=3469.71 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 919560 0 0 0 346522 449 0 0 25 0 5 0 521447088 4230729728 919495 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 1032893 919495 175 36 0 1030798 0
[pid=13035/tid=13037] ppid=13033 vsize=4131572 CPUtime=861.9 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 484328 0 0 0 85987 203 0 0 25 0 5 0 521447114 4230729728 919495 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=4131572 CPUtime=870.22 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 128796 0 0 0 86937 85 0 0 25 0 5 0 521447114 4230729728 919495 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=4131572 CPUtime=868.97 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 147497 0 0 0 86814 83 0 0 25 0 5 0 521447114 4230729728 919495 33554432000 4194304 4338100 140733263103008 18446744073709551615 242969945838 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=4131572 CPUtime=868.32 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 158031 0 0 0 86756 76 0 0 25 0 5 0 521447114 4230729728 919495 33554432000 4194304 4338100 140733263103008 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3469.71
Current children cumulated vsize (KiB) 4131572

[startup+942.301 s]
/proc/loadavg: 7.90 7.93 7.90 9/184 13096
/proc/meminfo: memFree=18366032/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=4325484 CPUtime=3704.07 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 974696 0 0 0 369927 480 0 0 25 0 5 0 521447088 4429295616 974631 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 1081371 974631 175 36 0 1079276 0
[pid=13035/tid=13037] ppid=13033 vsize=4325484 CPUtime=920.03 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 532633 0 0 0 91781 222 0 0 25 0 5 0 521447114 4429295616 974631 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=4325484 CPUtime=928.98 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 130853 0 0 0 92808 90 0 0 25 0 5 0 521447114 4429295616 974631 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=4325484 CPUtime=927.8 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 150427 0 0 0 92694 86 0 0 25 0 5 0 521447114 4429295616 974631 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=4325484 CPUtime=926.97 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 159875 0 0 0 92617 80 0 0 25 0 5 0 521447114 4429295616 974631 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3704.07
Current children cumulated vsize (KiB) 4325484

[startup+1002.3 s]
/proc/loadavg: 7.70 7.89 7.89 9/184 13098
/proc/meminfo: memFree=18068160/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=4469092 CPUtime=3937.43 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 1029358 0 0 0 393235 508 0 0 25 0 5 0 521447088 4576350208 1029293 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 1117273 1029293 175 36 0 1115178 0
[pid=13035/tid=13037] ppid=13033 vsize=4469092 CPUtime=977.79 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 568025 0 0 0 97542 237 0 0 25 0 5 0 521447114 4576350208 1029293 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=4469092 CPUtime=987.45 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 132777 0 0 0 98650 95 0 0 25 0 5 0 521447114 4576350208 1029293 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=4469092 CPUtime=986.39 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 159299 0 0 0 98549 90 0 0 25 0 5 0 521447114 4576350208 1029293 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=4469092 CPUtime=985.51 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 168349 0 0 0 98467 84 0 0 25 0 5 0 521447114 4576350208 1029293 33554432000 4194304 4338100 140733263103008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3937.43
Current children cumulated vsize (KiB) 4469092

[startup+1062.3 s]
/proc/loadavg: 7.99 7.93 7.91 9/184 13099
/proc/meminfo: memFree=17746100/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=4599412 CPUtime=4171.89 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 1067507 0 0 0 416655 534 0 0 25 0 5 0 521447088 4709797888 1067442 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 1149853 1067442 175 36 0 1147758 0
[pid=13035/tid=13037] ppid=13033 vsize=4599412 CPUtime=1036.02 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 600313 0 0 0 103349 253 0 0 25 0 5 0 521447114 4709797888 1067442 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=4599412 CPUtime=1046.19 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 134553 0 0 0 104520 99 0 0 25 0 5 0 521447114 4709797888 1067442 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=4599412 CPUtime=1044.76 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 161395 0 0 0 104383 93 0 0 25 0 5 0 521447114 4709797888 1067442 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=4599412 CPUtime=1044.65 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 170338 0 0 0 104377 88 0 0 25 0 5 0 521447114 4709797888 1067442 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4171.89
Current children cumulated vsize (KiB) 4599412

[startup+1122.3 s]
/proc/loadavg: 7.98 7.94 7.91 9/184 13101
/proc/meminfo: memFree=17475516/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=4616744 CPUtime=4407.04 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 1084710 0 0 0 440151 553 0 0 25 0 5 0 521447088 4727545856 1084645 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 1154186 1084645 175 36 0 1152091 0
[pid=13035/tid=13037] ppid=13033 vsize=4616744 CPUtime=1094.42 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 604432 0 0 0 109181 261 0 0 25 0 5 0 521447114 4727545856 1084645 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=4616744 CPUtime=1105.24 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 136473 0 0 0 110422 102 0 0 25 0 5 0 521447114 4727545856 1084645 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=4616744 CPUtime=1103.61 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 167801 0 0 0 110264 97 0 0 25 0 5 0 521447114 4727545856 1084645 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=4616744 CPUtime=1103.48 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 175096 0 0 0 110257 91 0 0 25 0 5 0 521447114 4727545856 1084645 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4407.04
Current children cumulated vsize (KiB) 4616744

[startup+1182.3 s]
/proc/loadavg: 8.17 8.00 7.93 9/184 13103
/proc/meminfo: memFree=17279696/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=4625716 CPUtime=4643.41 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) S 13033 13035 12928 0 -1 4202496 1099622 0 0 0 463769 572 0 0 25 0 5 0 521447088 4736733184 1099557 33554432000 4194304 4338100 140733263103008 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 1156429 1099557 175 36 0 1154334 0
[pid=13035/tid=13037] ppid=13033 vsize=4625716 CPUtime=1153.39 cores=1,3,5,7
/proc/13035/task/13037/stat : 13037 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 606120 0 0 0 115073 266 0 0 25 0 5 0 521447114 4736733184 1099557 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=13035/tid=13038] ppid=13033 vsize=4625716 CPUtime=1164.38 cores=1,3,5,7
/proc/13035/task/13038/stat : 13038 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 138286 0 0 0 116332 106 0 0 25 0 5 0 521447114 4736733184 1099557 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=13035/tid=13039] ppid=13033 vsize=4625716 CPUtime=1162.8 cores=1,3,5,7
/proc/13035/task/13039/stat : 13039 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 176295 0 0 0 116178 102 0 0 25 0 5 0 521447114 4736733184 1099557 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=13035/tid=13040] ppid=13033 vsize=4625716 CPUtime=1162.55 cores=1,3,5,7
/proc/13035/task/13040/stat : 13040 (ttsth-5-4) R 13033 13035 12928 0 -1 4202560 178013 0 0 0 116160 95 0 0 25 0 5 0 521447114 4736733184 1099557 33554432000 4194304 4338100 140733263103008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4643.41
Current children cumulated vsize (KiB) 4625716



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1222.11 s]
/proc/loadavg: 7.93 7.95 7.91 6/180 13104
/proc/meminfo: memFree=17961936/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=0 CPUtime=4800.11 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) R 13033 13035 12928 0 -1 4203524 1107343 0 0 0 479424 587 0 0 18 0 1 0 521447088 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1222.21 s]
/proc/loadavg: 7.93 7.95 7.91 6/180 13104
/proc/meminfo: memFree=17961936/32951132 swapFree=67111364/67111528
[pid=13035] ppid=13033 vsize=0 CPUtime=4800.2 cores=1,3,5,7
/proc/13035/stat : 13035 (ttsth-5-4) R 13033 13035 12928 0 -1 4203524 1107343 0 0 0 479424 596 0 0 18 0 1 0 521447088 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/13035/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1222.25
CPU time (s): 4800.26
CPU user time (s): 4794.25
CPU system time (s): 6.00909
CPU usage (%): 392.739
Max. virtual memory (cumulated for all children) (KiB): 4762300

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4794.25
system time used= 6.00909
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1107343
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 64981
involuntary context switches= 32168

runsolver used 1.93371 second user time and 4.36634 second system time

The end

Launcher Data 

Begin job on node128 at 2011-04-29 19:34:44
IDJOB=3263850
IDBENCH=84028
IDSOLVER=1654
FILE ID=node128/3263850-1304098484
RUNJOBID= node128-1304097280-12947
PBS_JOBID= 13170611
Free space on /tmp= 66972 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/stanion/hwb/hwb-n30-02-S77299857.shuffled-as.sat03-1633.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263850-1304098484/watcher-3263850-1304098484 -o /tmp/evaluation-result-3263850-1304098484/solver-3263850-1304098484 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263850-1304098484.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= bdf2c49c74bfdbd3bd4663dca402d24f
RANDOM SEED=1713142050

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      26625056 kB
Buffers:       1281980 kB
Cached:        4392008 kB
SwapCached:        160 kB
Active:         339416 kB
Inactive:      5400960 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      26625056 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:              68 kB
Writeback:           0 kB
AnonPages:       65936 kB
Mapped:          15372 kB
Slab:           523780 kB
PageTables:       4248 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   220164 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66972 MiB
End job on node128 at 2011-04-29 19:55:09