Trace number 3264114

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.16 1202.47

General information on the benchmark

NameSAT04/handmade/kexu/frb-0.6/
frb80-14-2.used-as.sat04-880.cnf
MD5SUM1242f0037df1c3d888415aeb418b77ed
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.696893
Satisfiable
(Un)Satisfiability was proved
Number of variables1120
Number of clauses43179
Sum of the clauses size87318
Maximum clause length14
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 243099
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 580

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264114-1304099035.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 1120
0.00/0.02	c Clauses = 43179
0.00/0.02	c Literals = 87318
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 1120
0.00/0.02	c Clauses = 43179
0.00/0.02	c Literals = 87318
0.00/0.02	c a2
0.00/0.05	c a1
0.00/0.09	c Independent sub-problems = 1
0.00/0.09	c Sub-problem with 1120 variables and 43179 clauses
0.00/0.09	c using permutation method 4
0.00/0.09	c count = 1
0.00/0.09	c Ordering finished 0.080 seconds
0.00/0.09	c mapped 0.090 seconds
0.00/0.09	c permuted 0.090 seconds
0.00/0.09	c about to sort
0.00/0.10	c sorted
0.10/0.18	c clauses now 43179
0.10/0.18	c init finished 0.100 seconds
0.10/0.18	c tree built 0.170 seconds
0.10/0.18	c 
0.10/0.18	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.10/0.18	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.10/0.18	c ................................ | ..........                    |                                  0.2         0.0             
0.10/0.18	c ................................ | ..........                  1 |                                  0.2         5.9             
0.10/0.18	c ................................ | ..........                  2 |                                  0.2        11.8             
0.10/0.18	c ................................ | ..........                  3 |                                  0.2        17.6             
0.10/0.18	c ................................ | ..........                  4 |                                  0.2        23.5             
0.10/0.18	c ................................ | ..........                  5 |                                  0.2        29.4             
0.10/0.18	c ................................ | ..........               1000 | 18           189                 0.2      5555.6         6158
95.71/24.17	c ................................ | ..........            2630000 | 171971       575521             96.0     27393.0     10867289
191.95/48.22	c ................................ | ..........            4447000 | 295440       998363            192.0     23157.8     15260387
287.76/72.26	c ................................ | ..........            5970000 | 382961       1325410           288.0     20727.7     19460672
383.99/96.31	c ................................ | ..........            7356000 | 468805       1608689           384.0     19154.3     24786862
479.82/120.35	c ................................ | *.........            8511000 | 545114       1852707           480.0     17730.1     27504813
576.04/144.40	c ................................ | *.........            9639000 | 621591       2068925           576.0     16732.9     29485053
671.92/168.44	c ................................ | *.........           10752000 | 694430       2293786           672.1     15997.9     31366464
767.75/192.47	c ................................ | *.........           11763000 | 764446       2538942           768.0     15315.6     33126533
863.96/216.52	c ................................ | *.........           13103000 | 842774       2819617           864.0     15164.8     36393431
959.76/240.57	c ................................ | **........           14428000 | 919102       3070238           960.0     15028.5     40212765
1055.96/264.62	c ................................ | **........           15626000 | 983652       3291969          1056.0     14796.6     42797105
1151.73/288.67	c ................................ | **........           16707000 | 1041943      3515465          1152.0     14502.1     46223376
1247.96/312.73	c ................................ | **........           17717000 | 1095133      3712059          1248.1     14195.3     50650134
1343.80/336.76	c ................................ | **........           18476000 | 1139929      3888295          1344.0     13746.7     51640144
1440.00/360.81	c ................................ | ***.......           19271000 | 1187143      4058211          1440.0     13382.5     54538744
1535.80/384.85	c ................................ | ***.......           20243000 | 1239682      4229649          1536.0     13179.0     57753652
1631.99/408.94	c ................................ | ***.......           21073000 | 1289453      4456191          1632.2     12911.2     60207139
1727.79/432.97	c ................................ | ***.......           21769000 | 1330986      4638235          1728.1     12597.1     62010970
1824.05/457.00	c ................................ | ***.......           22413000 | 1371104      4808332          1824.1     12287.4     64673909
1919.86/481.03	c ................................ | ****......           23053000 | 1409382      4955163          1920.0     12006.7     66192154
2015.72/505.09	c ................................ | ****......           23734000 | 1448790      5103425          2016.0     11772.5     67519165
2111.97/529.14	c ................................ | ****......           24425000 | 1492093      5252919          2112.1     11564.3     70214027
2207.78/553.17	c ................................ | ****......           25005000 | 1528934      5398479          2208.1     11324.2     74315321
2303.98/577.23	c ................................ | ****......           25661000 | 1568928      5534840          2304.1     11137.1     77858778
2399.78/601.26	c ................................ | *****.....           26635000 | 1618755      5699644          2400.1     11097.6     81652061
2496.03/625.30	c ................................ | *****.....           27574000 | 1666591      5890952          2496.0     11047.1     83511489
2591.81/649.37	c ................................ | *****.....           28492000 | 1710612      6073777          2592.1     10991.8     84919234
2688.01/673.41	c ................................ | *****.....           29328000 | 1754704      6240579          2688.1     10910.5     88180889
2783.89/697.46	c ................................ | *****.....           30079000 | 1792205      6383405          2784.1     10803.9     91126074
2879.70/721.47	c ................................ | ******....           30836000 | 1831426      6532013          2880.0     10706.9     93714346
2975.96/745.51	c ................................ | ******....           31559000 | 1870603      6677053          2976.0     10604.4     94055717
3071.81/769.57	c ................................ | ******....           32309000 | 1909678      6828864          3072.1     10516.8     95568080
3168.05/793.61	c ................................ | ******....           33126000 | 1960665      6962518          3168.1     10456.1     97785167
3263.86/817.66	c ................................ | ******....           33891000 | 2012631      7078171          3264.1     10382.9     97785167
3360.12/841.70	c ................................ | *******...           34647000 | 2065314      7203775          3360.1     10311.2     98598082
3455.95/865.72	c ................................ | *******...           35362000 | 2115090      7338641          3456.0     10231.9    100557347
3551.72/889.77	c ................................ | *******...           36034000 | 2163574      7450979          3552.0     10144.7    102367972
3647.92/913.83	c ................................ | *******...           36715000 | 2212293      7568417          3648.1     10064.2    103194865
3743.70/937.87	c ................................ | *******...           37409000 | 2260505      7730287          3744.0      9991.7    103762757
3839.90/961.93	c ................................ | ********..           38060000 | 2309849      7868718          3840.1      9911.3    103762757
3935.74/985.97	c ................................ | ********..           38822000 | 2357678      8014991          3936.0      9863.3    104691554
4031.99/1010.02	c ................................ | ********..           39647000 | 2406410      8159006          4032.1      9832.9    105278644
4127.79/1034.06	c ................................ | ********..           40311000 | 2447111      8295170          4128.0      9765.2    107521962
4224.04/1058.11	c ................................ | ********..           40899000 | 2481709      8416137          4224.1      9682.3    109781341
4319.83/1082.16	c ................................ | *********.           41449000 | 2515490      8524035          4320.1      9594.5    110430682
4416.07/1106.20	c ................................ | *********.           42149000 | 2555127      8650547          4416.1      9544.4    110430682
4511.92/1130.25	c ................................ | *********.           42758000 | 2587863      8763712          4512.1      9476.2    110629762
4607.79/1154.26	c ................................ | *********.           43276000 | 2620567      8856036          4608.1      9391.4    111450227
4703.67/1178.29	c ................................ | *********.           43813000 | 2652394      8951603          4704.0      9313.9    113015294
4799.93/1202.32	c CPU limit (4800) reached
4799.93/1202.32	c Time = 4800.02 seconds
4799.93/1202.32	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264114-1304099035/watcher-3264114-1304099035 -o /tmp/evaluation-result-3264114-1304099035/solver-3264114-1304099035 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264114-1304099035.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.43 7.97 7.99 2/174 5495
/proc/meminfo: memFree=23622028/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=9892 CPUtime=0 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) R 5493 5495 5016 0 -1 4202496 533 0 0 0 0 0 0 0 22 0 1 0 600134154 10129408 475 33554432000 4194304 4338100 140735535843872 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/5495/statm: 2473 476 147 36 0 378 0

[startup+0.104888 s]
/proc/loadavg: 7.43 7.97 7.99 2/174 5495
/proc/meminfo: memFree=23622028/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=18736 CPUtime=0.1 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) R 5493 5495 5016 0 -1 4202496 3016 0 0 0 10 0 0 0 23 0 1 0 600134154 19185664 2702 33554432000 4194304 4338100 140735535843872 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/5495/statm: 4684 2704 163 36 0 2589 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 18736

[startup+0.200811 s]
/proc/loadavg: 7.43 7.97 7.99 2/174 5495
/proc/meminfo: memFree=23622028/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=162636 CPUtime=0.25 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 9788 0 0 0 24 1 0 0 23 0 5 0 600134154 166539264 9388 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 40659 9390 176 36 0 38564 0
Current children cumulated CPU time (s) 0.25
Current children cumulated vsize (KiB) 162636

[startup+0.300794 s]
/proc/loadavg: 7.43 7.97 7.99 2/174 5495
/proc/meminfo: memFree=23622028/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=162900 CPUtime=0.64 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 10807 0 0 0 63 1 0 0 23 0 5 0 600134154 166809600 10407 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 40725 10409 176 36 0 38630 0
Current children cumulated CPU time (s) 0.64
Current children cumulated vsize (KiB) 162900

[startup+0.700721 s]
/proc/loadavg: 7.43 7.97 7.99 2/174 5495
/proc/meminfo: memFree=23622028/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=234824 CPUtime=2.25 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 13867 0 0 0 222 3 0 0 23 0 5 0 600134154 240459776 13467 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 58706 13469 176 36 0 56611 0
Current children cumulated CPU time (s) 2.25
Current children cumulated vsize (KiB) 234824

[startup+1.50057 s]
/proc/loadavg: 7.47 7.97 7.99 10/186 5510
/proc/meminfo: memFree=23477976/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=260112 CPUtime=5.44 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 19264 0 0 0 540 4 0 0 23 0 5 0 600134154 266354688 18864 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 65028 18864 176 36 0 62933 0
[pid=5495/tid=5503] ppid=5493 vsize=260112 CPUtime=1.31 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 3263 0 0 0 131 0 0 0 19 0 5 0 600134172 266354688 18864 33554432000 4194304 4338100 140735535843872 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=260112 CPUtime=1.3 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 1906 0 0 0 129 1 0 0 25 0 5 0 600134172 266354688 18864 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917206734 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=260112 CPUtime=1.31 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 3364 0 0 0 131 0 0 0 25 0 5 0 600134172 266354688 18865 33554432000 4194304 4338100 140735535843872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=260112 CPUtime=1.3 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) S 5493 5495 5016 0 -1 4202560 1509 0 0 0 130 0 0 0 25 0 5 0 600134172 266354688 18865 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917219700 0 0 4096 0 18446604471108815616 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 5.44
Current children cumulated vsize (KiB) 260112

[startup+3.10028 s]
/proc/loadavg: 7.47 7.97 7.99 9/186 5510
/proc/meminfo: memFree=23407800/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=301660 CPUtime=11.83 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 29157 0 0 0 1175 8 0 0 23 0 5 0 600134154 308899840 28757 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 75415 28759 176 36 0 73320 0
[pid=5495/tid=5503] ppid=5493 vsize=301660 CPUtime=2.91 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 6691 0 0 0 290 1 0 0 25 0 5 0 600134172 308899840 28759 33554432000 4194304 4338100 140735535843872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=301660 CPUtime=2.89 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 3765 0 0 0 287 2 0 0 25 0 5 0 600134172 308899840 28759 33554432000 4194304 4338100 140735535843872 18446744073709551615 260904789039 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=301660 CPUtime=2.91 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 6566 0 0 0 291 0 0 0 25 0 5 0 600134172 308899840 28759 33554432000 4194304 4338100 140735535843872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=301660 CPUtime=2.9 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 2914 0 0 0 288 2 0 0 25 0 5 0 600134172 308899840 28759 33554432000 4194304 4338100 140735535843872 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.83
Current children cumulated vsize (KiB) 301660

[startup+6.30069 s]
/proc/loadavg: 7.47 7.97 7.99 9/186 5510
/proc/meminfo: memFree=23222556/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=363504 CPUtime=24.61 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 44755 0 0 0 2441 20 0 0 23 0 5 0 600134154 372228096 44355 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 90876 44355 176 36 0 88781 0
[pid=5495/tid=5503] ppid=5493 vsize=363504 CPUtime=6.11 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 13101 0 0 0 608 3 0 0 25 0 5 0 600134172 372228096 44355 33554432000 4194304 4338100 140735535843872 18446744073709551615 4230687 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=363504 CPUtime=6.08 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 5954 0 0 0 601 7 0 0 25 0 5 0 600134172 372228096 44355 33554432000 4194304 4338100 140735535843872 18446744073709551615 260904789063 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=363504 CPUtime=6.11 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 11620 0 0 0 610 1 0 0 25 0 5 0 600134172 372228096 44355 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=363504 CPUtime=6.11 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 4857 0 0 0 604 7 0 0 25 0 5 0 600134172 372228096 44357 33554432000 4194304 4338100 140735535843872 18446744073709551615 260904789124 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 24.61
Current children cumulated vsize (KiB) 363504

[startup+12.7005 s]
/proc/loadavg: 7.55 7.97 7.99 9/186 5510
/proc/meminfo: memFree=22924360/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=455660 CPUtime=50.17 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 68446 0 0 0 4963 54 0 0 23 0 5 0 600134154 466595840 68046 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 113915 68048 176 36 0 111820 0
[pid=5495/tid=5503] ppid=5493 vsize=455660 CPUtime=12.51 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 22777 0 0 0 1246 5 0 0 25 0 5 0 600134172 466595840 68048 33554432000 4194304 4338100 140735535843872 18446744073709551615 260904789438 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=455660 CPUtime=12.44 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 9006 0 0 0 1221 23 0 0 25 0 5 0 600134172 466595840 68048 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=455660 CPUtime=12.51 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 20759 0 0 0 1248 3 0 0 25 0 5 0 600134172 466595840 68048 33554432000 4194304 4338100 140735535843872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=455660 CPUtime=12.51 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 6683 0 0 0 1230 21 0 0 25 0 5 0 600134172 466595840 68048 33554432000 4194304 4338100 140735535843872 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 50.17
Current children cumulated vsize (KiB) 455660

[startup+25.5012 s]
/proc/loadavg: 7.62 7.97 7.99 9/186 5511
/proc/meminfo: memFree=22495572/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=595288 CPUtime=101.29 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 103253 0 0 0 10009 120 0 0 23 0 5 0 600134154 609574912 102853 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 148822 102853 176 36 0 146727 0
[pid=5495/tid=5503] ppid=5493 vsize=595288 CPUtime=25.31 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 36356 0 0 0 2521 10 0 0 25 0 5 0 600134172 609574912 102853 33554432000 4194304 4338100 140735535843872 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=595288 CPUtime=25.16 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 12677 0 0 0 2466 50 0 0 25 0 5 0 600134172 609574912 102853 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=595288 CPUtime=25.31 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 34241 0 0 0 2525 6 0 0 25 0 5 0 600134172 609574912 102853 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=595288 CPUtime=25.3 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 10756 0 0 0 2479 51 0 0 25 0 5 0 600134172 609574912 102853 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 101.29
Current children cumulated vsize (KiB) 595288

[startup+51.1005 s]
/proc/loadavg: 7.77 7.97 7.99 9/186 5511
/proc/meminfo: memFree=21890404/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=752320 CPUtime=203.53 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 143462 0 0 0 20095 258 0 0 23 0 5 0 600134154 770375680 143062 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 188080 143062 176 36 0 185985 0
[pid=5495/tid=5503] ppid=5493 vsize=752320 CPUtime=50.79 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 55032 0 0 0 5062 17 0 0 25 0 5 0 600134172 770375680 143062 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=752320 CPUtime=50.72 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 17618 0 0 0 4957 115 0 0 25 0 5 0 600134172 770375680 143062 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=752320 CPUtime=50.9 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 47745 0 0 0 5081 9 0 0 25 0 5 0 600134172 770375680 143062 33554432000 4194304 4338100 140735535843872 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=752320 CPUtime=50.9 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 13844 0 0 0 4976 114 0 0 25 0 5 0 600134172 770375680 143062 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 203.53
Current children cumulated vsize (KiB) 752320

[startup+102.301 s]
/proc/loadavg: 7.90 7.97 7.99 9/186 5513
/proc/meminfo: memFree=20974904/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=1073036 CPUtime=407.94 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 222031 0 0 0 40287 507 0 0 23 0 5 0 600134154 1098788864 221631 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 268259 221631 176 36 0 266164 0
[pid=5495/tid=5503] ppid=5493 vsize=1073036 CPUtime=101.94 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 89339 0 0 0 10162 32 0 0 25 0 5 0 600134172 1098788864 221631 33554432000 4194304 4338100 140735535843872 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=1073036 CPUtime=101.89 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 26961 0 0 0 9962 227 0 0 25 0 5 0 600134172 1098788864 221631 33554432000 4194304 4338100 140735535843872 18446744073709551615 260905090480 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=1073036 CPUtime=101.83 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 71199 0 0 0 10167 16 0 0 25 0 5 0 600134172 1098788864 221631 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=1073036 CPUtime=102.07 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 25309 0 0 0 9978 229 0 0 25 0 5 0 600134172 1098788864 221631 33554432000 4194304 4338100 140735535843872 18446744073709551615 260904789438 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 407.94
Current children cumulated vsize (KiB) 1073036

[startup+162.301 s]
/proc/loadavg: 8.04 8.00 8.00 9/186 5514
/proc/meminfo: memFree=20237188/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=1252440 CPUtime=647.54 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 267806 0 0 0 64016 738 0 0 23 0 5 0 600134154 1282498560 267406 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 313110 267406 176 36 0 311015 0
[pid=5495/tid=5503] ppid=5493 vsize=1252440 CPUtime=161.92 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 114597 0 0 0 16148 44 0 0 25 0 5 0 600134172 1282498560 267406 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=1252440 CPUtime=161.87 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 27743 0 0 0 15855 332 0 0 25 0 5 0 600134172 1282498560 267406 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=1252440 CPUtime=161.49 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 89901 0 0 0 16128 21 0 0 25 0 5 0 600134172 1282498560 267406 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=1252440 CPUtime=162.04 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 26342 0 0 0 15867 337 0 0 25 0 5 0 600134172 1282498560 267406 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 647.54
Current children cumulated vsize (KiB) 1252440


################
# More data... #
################

/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 66116 0 0 0 68886 1217 0 0 25 0 5 0 600134172 3259375616 757390 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2803.43
Current children cumulated vsize (KiB) 3182984

[startup+762.301 s]
/proc/loadavg: 8.08 8.09 8.02 9/186 5530
/proc/meminfo: memFree=14678600/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=3289108 CPUtime=3043.08 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 783093 0 0 0 301490 2818 0 0 23 0 5 0 600134154 3368046592 782693 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 822277 782693 176 36 0 820182 0
[pid=5495/tid=5503] ppid=5493 vsize=3289108 CPUtime=761.42 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 380826 0 0 0 75981 161 0 0 25 0 5 0 600134172 3368046592 782693 33554432000 4194304 4338100 140735535843872 18446744073709551615 4245205 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=3289108 CPUtime=760.98 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 64009 0 0 0 74809 1289 0 0 25 0 5 0 600134172 3368046592 782693 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=3289108 CPUtime=759.45 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 259289 0 0 0 75872 73 0 0 25 0 5 0 600134172 3368046592 782693 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=3289108 CPUtime=761.02 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 69746 0 0 0 74810 1292 0 0 25 0 5 0 600134172 3368046592 782693 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3043.08
Current children cumulated vsize (KiB) 3289108

[startup+822.301 s]
/proc/loadavg: 8.03 8.07 8.02 9/186 5532
/proc/meminfo: memFree=14336348/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=3397504 CPUtime=3282.63 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 811612 0 0 0 325296 2967 0 0 23 0 5 0 600134154 3479044096 811212 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 849376 811212 176 36 0 847281 0
[pid=5495/tid=5503] ppid=5493 vsize=3397504 CPUtime=821.38 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 396648 0 0 0 81969 169 0 0 25 0 5 0 600134172 3479044096 811212 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=3397504 CPUtime=820.96 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 65443 0 0 0 80741 1355 0 0 25 0 5 0 600134172 3479044096 811212 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=3397504 CPUtime=819.08 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 269473 0 0 0 81831 77 0 0 25 0 5 0 600134172 3479044096 811212 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=3397504 CPUtime=821.01 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 70825 0 0 0 80737 1364 0 0 25 0 5 0 600134172 3479044096 811212 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3282.63
Current children cumulated vsize (KiB) 3397504

[startup+882.307 s]
/proc/loadavg: 8.05 8.07 8.02 9/186 5533
/proc/meminfo: memFree=14123288/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=3530200 CPUtime=3522.23 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 845928 0 0 0 349097 3126 0 0 23 0 5 0 600134154 3614924800 845528 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 882550 845528 176 36 0 880455 0
[pid=5495/tid=5503] ppid=5493 vsize=3530200 CPUtime=881.34 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 414606 0 0 0 87953 181 0 0 25 0 5 0 600134172 3614924800 845528 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=3530200 CPUtime=880.8 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 65610 0 0 0 86654 1426 0 0 25 0 5 0 600134172 3614924800 845528 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=3530200 CPUtime=878.88 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 285374 0 0 0 87805 83 0 0 25 0 5 0 600134172 3614924800 845528 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=3530200 CPUtime=880.99 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 71115 0 0 0 86666 1433 0 0 25 0 5 0 600134172 3614924800 845528 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3522.23
Current children cumulated vsize (KiB) 3530200

[startup+942.301 s]
/proc/loadavg: 8.05 8.07 8.02 9/186 5535
/proc/meminfo: memFree=13897964/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=3625656 CPUtime=3761.65 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 868307 0 0 0 372867 3298 0 0 23 0 5 0 600134154 3712671744 867907 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 906414 867907 176 36 0 904319 0
[pid=5495/tid=5503] ppid=5493 vsize=3625656 CPUtime=941.27 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 422428 0 0 0 93941 186 0 0 25 0 5 0 600134172 3712671744 867907 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=3625656 CPUtime=940.41 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 68744 0 0 0 92533 1508 0 0 25 0 5 0 600134172 3712671744 867907 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=3625656 CPUtime=938.81 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 293678 0 0 0 93796 85 0 0 25 0 5 0 600134172 3712671744 867907 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=3625656 CPUtime=940.94 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 74234 0 0 0 92579 1515 0 0 25 0 5 0 600134172 3712671744 867907 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3761.65
Current children cumulated vsize (KiB) 3625656

[startup+1002.3 s]
/proc/loadavg: 8.05 8.07 8.01 9/186 5537
/proc/meminfo: memFree=13711680/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=3673896 CPUtime=4001.24 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 881047 0 0 0 396652 3472 0 0 23 0 5 0 600134154 3762069504 880647 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 918474 880647 176 36 0 916379 0
[pid=5495/tid=5503] ppid=5493 vsize=3673896 CPUtime=1001.23 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 431648 0 0 0 99930 193 0 0 25 0 5 0 600134172 3762069504 880647 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=3673896 CPUtime=1000.41 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 68960 0 0 0 98449 1592 0 0 25 0 5 0 600134172 3762069504 880647 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=3673896 CPUtime=998.79 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 296796 0 0 0 99792 87 0 0 25 0 5 0 600134172 3762069504 880647 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=3673896 CPUtime=1000.6 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 74420 0 0 0 98462 1598 0 0 25 0 5 0 600134172 3762069504 880647 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4001.24
Current children cumulated vsize (KiB) 3673896

[startup+1062.3 s]
/proc/loadavg: 8.14 8.08 8.02 9/186 5547
/proc/meminfo: memFree=13486496/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=3828356 CPUtime=4240.79 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 920233 0 0 0 420459 3620 0 0 23 0 5 0 600134154 3920236544 919833 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 957089 919833 176 36 0 954994 0
[pid=5495/tid=5503] ppid=5493 vsize=3828356 CPUtime=1061.08 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 450301 0 0 0 105905 203 0 0 25 0 5 0 600134172 3920236544 919833 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=3828356 CPUtime=1060.36 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 69179 0 0 0 104379 1657 0 0 25 0 5 0 600134172 3920236544 919833 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=3828356 CPUtime=1058.6 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 316896 0 0 0 105767 93 0 0 25 0 5 0 600134172 3920236544 919833 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=3828356 CPUtime=1060.54 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 74634 0 0 0 104390 1664 0 0 25 0 5 0 600134172 3920236544 919833 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4240.79
Current children cumulated vsize (KiB) 3828356

[startup+1122.3 s]
/proc/loadavg: 8.16 8.10 8.02 10/186 5549
/proc/meminfo: memFree=13302932/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=3838928 CPUtime=4480.36 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 927266 0 0 0 444282 3754 0 0 23 0 5 0 600134154 3931062272 926866 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 959732 926866 176 36 0 957637 0
[pid=5495/tid=5503] ppid=5493 vsize=3838928 CPUtime=1120.89 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 451736 0 0 0 111884 205 0 0 25 0 5 0 600134172 3931062272 926866 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=3838928 CPUtime=1120.34 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 71631 0 0 0 110312 1722 0 0 25 0 5 0 600134172 3931062272 926866 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=3838928 CPUtime=1118.41 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 317938 0 0 0 111747 94 0 0 25 0 5 0 600134172 3931062272 926866 33554432000 4194304 4338100 140735535843872 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=3838928 CPUtime=1120.52 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 76738 0 0 0 110321 1731 0 0 25 0 5 0 600134172 3931062272 926866 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917220021 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4480.36
Current children cumulated vsize (KiB) 3838928

[startup+1182.3 s]
/proc/loadavg: 8.23 8.12 8.03 9/186 5550
/proc/meminfo: memFree=13102540/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=3927304 CPUtime=4720.02 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) S 5493 5495 5016 0 -1 4202496 950823 0 0 0 468100 3902 0 0 23 0 5 0 600134154 4021559296 950423 33554432000 4194304 4338100 140735535843872 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 981826 950423 176 36 0 979731 0
[pid=5495/tid=5503] ppid=5493 vsize=3927304 CPUtime=1180.56 cores=1,3,5,7
/proc/5495/task/5503/stat : 5503 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 470419 0 0 0 117840 216 0 0 25 0 5 0 600134172 4021559296 950423 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=5495/tid=5504] ppid=5493 vsize=3927304 CPUtime=1180.33 cores=1,3,5,7
/proc/5495/task/5504/stat : 5504 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 71835 0 0 0 116243 1790 0 0 25 0 5 0 600134172 4021559296 950423 33554432000 4194304 4338100 140735535843872 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=3927304 CPUtime=1178.41 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 322421 0 0 0 117745 96 0 0 25 0 5 0 600134172 4021559296 950423 33554432000 4194304 4338100 140735535843872 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=5495/tid=5506] ppid=5493 vsize=3927304 CPUtime=1180.51 cores=1,3,5,7
/proc/5495/task/5506/stat : 5506 (ttsth-5-4) R 5493 5495 5016 0 -1 4202560 76925 0 0 0 116254 1797 0 0 25 0 5 0 600134172 4021559296 950423 33554432000 4194304 4338100 140735535843872 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4720.02
Current children cumulated vsize (KiB) 3927304



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.41 s]
/proc/loadavg: 8.17 8.11 8.03 6/183 5551
/proc/meminfo: memFree=15117232/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) Z 5493 5495 5016 0 -1 4203548 959260 0 0 0 476057 3953 0 0 18 0 2 0 600134154 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 0 0 0 0 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=0 CPUtime=1198.52 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4203588 326446 0 0 0 119745 107 0 0 25 0 2 0 600134172 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.41 s]
/proc/loadavg: 8.17 8.11 8.03 6/183 5551
/proc/meminfo: memFree=15117232/32950928 swapFree=67071984/67111528
[pid=5495] ppid=5493 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/5495/stat : 5495 (ttsth-5-4) Z 5493 5495 5016 0 -1 4203548 959260 0 0 0 476057 3953 0 0 18 0 2 0 600134154 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/5495/statm: 0 0 0 0 0 0 0
[pid=5495/tid=5505] ppid=5493 vsize=0 CPUtime=1198.52 cores=1,3,5,7
/proc/5495/task/5505/stat : 5505 (ttsth-5-4) R 5493 5495 5016 0 -1 4203588 326446 0 0 0 119745 107 0 0 25 0 2 0 600134172 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.47
CPU time (s): 4800.16
CPU user time (s): 4760.58
CPU system time (s): 39.587
CPU usage (%): 399.193
Max. virtual memory (cumulated for all children) (KiB): 3965236

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4760.58
system time used= 39.587
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 959260
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5918
involuntary context switches= 20046

runsolver used 2.12568 second user time and 4.45032 second system time

The end

Launcher Data 

Begin job on node116 at 2011-04-29 19:43:55
IDJOB=3264114
IDBENCH=84554
IDSOLVER=1654
FILE ID=node116/3264114-1304099035
RUNJOBID= node116-1304099035-5482
PBS_JOBID= 13170597
Free space on /tmp= 72516 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/kexu/frb-0.6/frb80-14-2.used-as.sat04-880.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264114-1304099035/watcher-3264114-1304099035 -o /tmp/evaluation-result-3264114-1304099035/solver-3264114-1304099035 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264114-1304099035.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1242f0037df1c3d888415aeb418b77ed
RANDOM SEED=1054548790

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23622556 kB
Buffers:       1904868 kB
Cached:        6952696 kB
SwapCached:       9732 kB
Active:        4401108 kB
Inactive:      4498176 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23622556 kB
SwapTotal:    67111528 kB
SwapFree:     67071984 kB
Dirty:             532 kB
Writeback:           0 kB
AnonPages:       35604 kB
Mapped:          13492 kB
Slab:           365820 kB
PageTables:       4740 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   189112 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72512 MiB
End job on node116 at 2011-04-29 20:03:59