iSharkFly-Open
/
VistA-fmts
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

creating logs directory for file store load test results

This commit is contained in:
george 2011-11-13 07:00:53 +00:00
parent c543ee1a1f
commit cecde026df
6 changed files with 2984 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

103
logs/F2N-benchmarks.txt Normal file
View File

@ -0,0 +1,103 @@
GTM>D IMPORT^C0XMAIN("smart-rdf-in/collins-frank.rdf")
STARTED: 3111104.130509
READING IN: smart-rdf-in/collins-frank.rdf
200 LINES READ
ADDED: _:G072744409 _S:795646155 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/collins-frank_rdf_585185215
128 XML NODES PARSED
INSERTING 99 TRIPLES
ENDED AT: 3111104.13051
ELAPSED TIME: 1 SECONDS
APPROXIMATELY 99 TRIPLES PER SECOND
GTM>D IMPORT^C0XMAIN("smart-rdf-in/reed-richard.rdf")
STARTED: 3111104.130606
READING IN: smart-rdf-in/reed-richard.rdf
722 LINES READ
ADDED: _:G758268243 _S:177410100 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/reed-richard_rdf_213828749
462 XML NODES PARSED
INSERTING 339 TRIPLES
ENDED AT: 3111104.13061
ELAPSED TIME: 4 SECONDS
APPROXIMATELY 84 TRIPLES PER SECOND
GTM>D IMPORT^C0XMAIN("smart-rdf-in/cole-susan.rdf")
STARTED: 3111104.130628
READING IN: smart-rdf-in/cole-susan.rdf
3428 LINES READ
ADDED: _:G271187746 _S:899679576 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/cole-susan_rdf_538236597
2101 XML NODES PARSED
SKIPPING NODE: 9
SKIPPING NODE: 26
SKIPPING NODE: 29
SKIPPING NODE: 32
SKIPPING NODE: 35
INSERTING 1425 TRIPLES
ENDED AT: 3111104.130645
ELAPSED TIME: 17 SECONDS
APPROXIMATELY 83 TRIPLES PER SECOND
GTM>D IMPORT^C0XMAIN("smart-rdf-in/ford-shirley.rdf")
STARTED: 3111104.130703
READING IN: smart-rdf-in/ford-shirley.rdf
8922 LINES READ
ADDED: _:G740421472 _S:922849860 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/ford-shirley_rdf_809878775
5470 XML NODES PARSED
ERROR, NO OBJECT FOUND FOR NODE: 4217
ERROR, NO OBJECT FOUND FOR NODE: 4226
ERROR, NO OBJECT FOUND FOR NODE: 4232
ERROR, NO OBJECT FOUND FOR NODE: 4258
ERROR, NO OBJECT FOUND FOR NODE: 4267
ERROR, NO OBJECT FOUND FOR NODE: 4273
INSERTING 3745 TRIPLES
ENDED AT: 3111104.130756
ELAPSED TIME: 53 SECONDS
APPROXIMATELY 70 TRIPLES PER SECOND
GTM>D IMPORT^C0XMAIN("smart-rdf-in/gracia-paul.rdf")
STARTED: 3111104.130817
READING IN: smart-rdf-in/gracia-paul.rdf
10698 LINES READ
ADDED: _:G289354757 _S:026395070 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/gracia-paul_rdf_297957633
6571 XML NODES PARSED
ERROR, NO OBJECT FOUND FOR NODE: 2751
ERROR, NO OBJECT FOUND FOR NODE: 2760
ERROR, NO OBJECT FOUND FOR NODE: 2766
ERROR, NO OBJECT FOUND FOR NODE: 6329
INSERTING 4512 TRIPLES
ENDED AT: 3111104.130928
ELAPSED TIME: 71 SECONDS
APPROXIMATELY 63 TRIPLES PER SECOND
GTM>D IMPORT^C0XF2N("qds/qds.rdf")
STARTED: 3111104.152012
READING IN: qds/qds.rdf
73528 LINES READ
ADDED: _:G951670203 _S:805831840 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/ge
o
rge/fmts/trunk/samples/qds/qds_rdf_394416761
71530 XML NODES PARSED
PARSE COMPLETE AT 3111104.152023
ELAPSED TIME: 8 SECONDS
APPROXIMATELY 8941 NODES PER SECOND
TRIPLES COMPLETE AT 3111104.153436
ELAPSED TIME: 853 SECONDS
APPROXIMATELY 81 TRIPLES PER SECOND <------ this calculation is wrong, because "batches" of 10000 nodes at a time were processed mixed in with the dom node->triple processing
INSERTING 69537 TRIPLES
INSERTION COMPLETE AT 3111104.152023
ELAPSED TIME: 127 SECONDS
APPROXIMATELY 547 NODES PER SECOND <------ I only wish that this were true, but alas it is based on the time to process the last "batch" divided by the total number of triples
ENDED AT: 3111104.153643
ELAPSED TIME: 991 SECONDS
APPROXIMATELY 70 TRIPLES PER SECOND <----- this "overall" rate is the one to watch...

95
logs/F2N-gpl-benchmark.txt Normal file
View File

@ -0,0 +1,95 @@
*************************
benchmark of F2N qds load on gpl.mdc-crew.net with GTM version 5.4-001_x8664 and MM turned off
****************************
GTM>D WGET^C0XF2N("https://trac.opensourcevista.net/svn/qrda/qds/qds.rdf")
STARTED: 3111104.140909
DOWNLOADING: https://trac.opensourcevista.net/svn/qrda/qds/qds.rdf
73944 LINES READ
DOWNLOAD COMPLETE AT 3111104.140936
ELAPSED TIME: 27 SECONDS
APPROXIMATELY 2738 LINES PER SEC
ADDED: _:G388521152 _S:326561669 fmts:rdfSource _TXT_INCOMING_RDF_FILE_https://trac.opensourcevista.net/svn/qrda/qds/qds.rdf_617430774
71530 XML NODES PARSED
PARSE COMPLETE AT 3111104.14102
ELAPSED TIME: 34 SECONDS
APPROXIMATELY 2103 NODES PER SECOND
TRIPLES COMPLETE AT 3111104.143302
ELAPSED TIME: 1362 SECONDS
APPROXIMATELY 51 TRIPLES PER SECOND <--- bogus
INSERTING 69537 TRIPLES
INSERTION COMPLETE AT 3111104.14102
ELAPSED TIME: 248 SECONDS
APPROXIMATELY 280 NODES PER SECOND <--- bogus
ENDED AT: 3111104.14371
ELAPSED TIME: 1681 SECONDS
APPROXIMATELY 41 TRIPLES PER SECOND < --- correct overall rate
*************************
benchmark of F2N qds load on gpl.mdc-crew.net with GTM version 5.4-001_x8664 and MM turned on
****************************
GTM>D WGET^C0XF2N("https://trac.opensourcevista.net/svn/qrda/qds/qds.rdf")
STARTED: 3111104.14521
DOWNLOADING: https://trac.opensourcevista.net/svn/qrda/qds/qds.rdf
73944 LINES READ
DOWNLOAD COMPLETE AT 3111104.145231
ELAPSED TIME: 21 SECONDS
APPROXIMATELY 3521 LINES PER SEC
ADDED: _:G356052990 _S:119674668 fmts:rdfSource _TXT_INCOMING_RDF_FILE_https://t
rac.opensourcevista.net/svn/qrda/qds/qds.rdf_073921057
71530 XML NODES PARSED
PARSE COMPLETE AT 3111104.1453
ELAPSED TIME: 21 SECONDS
APPROXIMATELY 3406 NODES PER SECOND
TRIPLES COMPLETE AT 3111104.151902
ELAPSED TIME: 1562 SECONDS
APPROXIMATELY 44 TRIPLES PER SECOND <-- bogus
INSERTING 69537 TRIPLES
INSERTION COMPLETE AT 3111104.1453
ELAPSED TIME: 223 SECONDS
APPROXIMATELY 311 NODES PER SECOND <-- bogus
ENDED AT: 3111104.152245
ELAPSED TIME: 1835 SECONDS
APPROXIMATELY 37 TRIPLES PER SECOND <-- correct overall rate
*************************
benchmark of F2N qds load on gpl.mdc-crew.net with GTM version 5.4-002B_x8664 and MM turned on
****************************
GTM>D WGET^C0XF2N("https://trac.opensourcevista.net/svn/fmts/trunk/samples/qds/q
ds.rdf")
STARTED: 3111104.170957
DOWNLOADING: https://trac.opensourcevista.net/svn/fmts/trunk/samples/qds/qds.rdf
73944 LINES READ
DOWNLOAD COMPLETE AT 3111104.171013
ELAPSED TIME: 16 SECONDS
APPROXIMATELY 4621 LINES PER SEC
ADDED: _:G310877834 _S:868017911 fmts:rdfSource _TXT_INCOMING_RDF_FILE_https://t
rac.opensourcevista.net/svn/fmts/trunk/samples/qds/qds.rdf_284111764
71530 XML NODES PARSED
PARSE COMPLETE AT 3111104.171039
ELAPSED TIME: 17 SECONDS
APPROXIMATELY 4207 NODES PER SECOND
TRIPLES COMPLETE AT 3111104.173038
ELAPSED TIME: 1199 SECONDS
APPROXIMATELY 57 TRIPLES PER SECOND
INSERTING 69537 TRIPLES
INSERTION COMPLETE AT 3111104.171039
ELAPSED TIME: 192 SECONDS
APPROXIMATELY 362 NODES PER SECOND
ENDED AT: 3111104.17335
APPROXIMATELY 362 NODES PER SECOND
ENDED AT: 3111104.17335
ELAPSED TIME: 1433 SECONDS
APPROXIMATELY 48 TRIPLES PER SECOND <== this is the interesting number, up from 37 tps

138
logs/F2N-raven-benchmarks.txt Normal file
View File

@ -0,0 +1,138 @@
GTM>D IMPORT^C0XMAIN("smart-rdf-in/collins-frank.rdf")
STARTED: 3111104.130509
READING IN: smart-rdf-in/collins-frank.rdf
200 LINES READ
ADDED: _:G072744409 _S:795646155 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/collins-frank_rdf_585185215
128 XML NODES PARSED
INSERTING 99 TRIPLES
ENDED AT: 3111104.13051
ELAPSED TIME: 1 SECONDS
APPROXIMATELY 99 TRIPLES PER SECOND
GTM>D IMPORT^C0XMAIN("smart-rdf-in/reed-richard.rdf")
STARTED: 3111104.130606
READING IN: smart-rdf-in/reed-richard.rdf
722 LINES READ
ADDED: _:G758268243 _S:177410100 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/reed-richard_rdf_213828749
462 XML NODES PARSED
INSERTING 339 TRIPLES
ENDED AT: 3111104.13061
ELAPSED TIME: 4 SECONDS
APPROXIMATELY 84 TRIPLES PER SECOND
GTM>D IMPORT^C0XMAIN("smart-rdf-in/cole-susan.rdf")
STARTED: 3111104.130628
READING IN: smart-rdf-in/cole-susan.rdf
3428 LINES READ
ADDED: _:G271187746 _S:899679576 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/cole-susan_rdf_538236597
2101 XML NODES PARSED
SKIPPING NODE: 9
SKIPPING NODE: 26
SKIPPING NODE: 29
SKIPPING NODE: 32
SKIPPING NODE: 35
INSERTING 1425 TRIPLES
ENDED AT: 3111104.130645
ELAPSED TIME: 17 SECONDS
APPROXIMATELY 83 TRIPLES PER SECOND
GTM>D IMPORT^C0XMAIN("smart-rdf-in/ford-shirley.rdf")
STARTED: 3111104.130703
READING IN: smart-rdf-in/ford-shirley.rdf
8922 LINES READ
ADDED: _:G740421472 _S:922849860 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/ford-shirley_rdf_809878775
5470 XML NODES PARSED
ERROR, NO OBJECT FOUND FOR NODE: 4217
ERROR, NO OBJECT FOUND FOR NODE: 4226
ERROR, NO OBJECT FOUND FOR NODE: 4232
ERROR, NO OBJECT FOUND FOR NODE: 4258
ERROR, NO OBJECT FOUND FOR NODE: 4267
ERROR, NO OBJECT FOUND FOR NODE: 4273
INSERTING 3745 TRIPLES
ENDED AT: 3111104.130756
ELAPSED TIME: 53 SECONDS
APPROXIMATELY 70 TRIPLES PER SECOND
GTM>D IMPORT^C0XMAIN("smart-rdf-in/gracia-paul.rdf")
STARTED: 3111104.130817
READING IN: smart-rdf-in/gracia-paul.rdf
10698 LINES READ
ADDED: _:G289354757 _S:026395070 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/smart-rdf-in/gracia-paul_rdf_297957633
6571 XML NODES PARSED
ERROR, NO OBJECT FOUND FOR NODE: 2751
ERROR, NO OBJECT FOUND FOR NODE: 2760
ERROR, NO OBJECT FOUND FOR NODE: 2766
ERROR, NO OBJECT FOUND FOR NODE: 6329
INSERTING 4512 TRIPLES
ENDED AT: 3111104.130928
ELAPSED TIME: 71 SECONDS
APPROXIMATELY 63 TRIPLES PER SECOND
*************************
benchmark of F2N qds load on raven with GT.M V5.4-002B Linux x86_64 and MM turned off
****************************
GTM>D IMPORT^C0XF2N("qds/qds.rdf")
STARTED: 3111104.152012
READING IN: qds/qds.rdf
73528 LINES READ
ADDED: _:G951670203 _S:805831840 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/ge
o
rge/fmts/trunk/samples/qds/qds_rdf_394416761
71530 XML NODES PARSED
PARSE COMPLETE AT 3111104.152023
ELAPSED TIME: 8 SECONDS
APPROXIMATELY 8941 NODES PER SECOND
TRIPLES COMPLETE AT 3111104.153436
ELAPSED TIME: 853 SECONDS
APPROXIMATELY 81 TRIPLES PER SECOND <------ this calculation is wrong, because "batches" of 10000 nodes at a time were processed mixed in with the dom node->triple processing
INSERTING 69537 TRIPLES
INSERTION COMPLETE AT 3111104.152023
ELAPSED TIME: 127 SECONDS
APPROXIMATELY 547 NODES PER SECOND <------ I only wish that this were true, but alas it is based on the time to process the last "batch" divided by the total number of triples
ENDED AT: 3111104.153643
ELAPSED TIME: 991 SECONDS
APPROXIMATELY 70 TRIPLES PER SECOND <----- this "overall" rate is the one to watch...
*************************
benchmark of F2N qds load on raven with GT.M V5.4-002B Linux x86_64 and MM turned on
****************************
GTM>D IMPORT^C0XF2N("qds/qds.rdf")
STARTED: 3111104.175524
READING IN: qds/qds.rdf
73528 LINES READ
ADDED: _:G802962321 _S:387624835 fmts:rdfSource _TXT_INCOMING_RDF_FILE_/home/geo
rge/fmts/trunk/samples/qds/qds_rdf_405517303
71530 XML NODES PARSED
PARSE COMPLETE AT 3111104.175535
ELAPSED TIME: 8 SECONDS
APPROXIMATELY 8941 NODES PER SECOND
TRIPLES COMPLETE AT 3111104.180951
ELAPSED TIME: 856 SECONDS
APPROXIMATELY 81 TRIPLES PER SECOND
INSERTING 69537 TRIPLES
INSERTION COMPLETE AT 3111104.175535
ELAPSED TIME: 130 SECONDS
APPROXIMATELY 534 NODES PER SECOND
ENDED AT: 3111104.181201
ELAPSED TIME: 997 SECONDS
APPROXIMATELY 69 TRIPLES PER SECOND

1344
logs/init-log-1.txt Normal file
View File

File diff suppressed because it is too large Load Diff

1171
logs/init-log-2.txt Normal file
View File

File diff suppressed because it is too large Load Diff

133
logs/speed-test.txt Normal file
View File

@ -0,0 +1,133 @@
GT.M Rocks!!!
gpl
I wrote a standalone test driver that will work on any GT.M system.
( you have to change the calls to $$NOW^XLFDT to another way to get a timestamp..
and then do the number of seconds computation differently...I don't know how)
I'm attaching the mumps routine C0XTEST.m
The first time I ran it I got significantly worse results. Then I realized that I had
generated a random Graph number for each node.... In our triples load all the
Graph numbers are the same... Here's what that first run looked like:
GTM>D EN^C0XTEST
INSERTION STARTS AT 3111105.234543
USING BULKLOAD
LOCKING ZERO NODE
ZERO NODE UNLOCKED, IENS RESERVED=10000
3111105.234543
INSERTION ENDS AT 3111106.000401
ELAPSED TIME: 1098 SECONDS
APPROXIMATELY 9 NODES PER SECOND
Second run on gpl (which is a VM) got worse, even though all the Graph pointer are the same.
GTM>D EN^C0XTEST
INSERTION STARTS AT 3111106.0032
USING BULKLOAD
LOCKING ZERO NODE
ZERO NODE UNLOCKED, IENS RESERVED=10000
3111106.0032
INSERTION ENDS AT 3111106.00532
ELAPSED TIME: 1280 SECONDS
APPROXIMATELY 7 NODES PER SECOND
I put C0XTEST on raven and I couldn't believe my eyes....
GTM>D EN^C0XTEST
INSERTION STARTS AT 3111106.024924
USING BULKLOAD
LOCKING ZERO NODE
ZERO NODE UNLOCKED, IENS RESERVED=10000
3111106.024924
INSERTION ENDS AT 3111106.024924
ELAPSED TIME: 0 SECONDS
GTM>W ^C0X(101,0)
C0X TRIPLE^172.101I^238634^238634
GTM>D EN^C0XTEST
INSERTION STARTS AT 3111106.024954
USING BULKLOAD
LOCKING ZERO NODE
ZERO NODE UNLOCKED, IENS RESERVED=10000
3111106.024954
INSERTION ENDS AT 3111106.024955
ELAPSED TIME: 1 SECONDS
APPROXIMATELY 10000 NODES PER SECOND
GTM>W ^C0X(101,0)
C0X TRIPLE^172.101I^248635^248635
GTM>D EN^C0XTEST
INSERTION STARTS AT 3111106.025009
USING BULKLOAD
LOCKING ZERO NODE
ZERO NODE UNLOCKED, IENS RESERVED=10000
3111106.025009
INSERTION ENDS AT 3111106.02501
ELAPSED TIME: 1 SECONDS
APPROXIMATELY 10000 NODES PER SECOND
GTM>W ^C0X(101,0)
C0X TRIPLE^172.101I^258636^258636
So I upped the anti to 100000 nodes
GTM>D EN^C0XTEST
INSERTION STARTS AT 3111106.025823
USING BULKLOAD
LOCKING ZERO NODE
ZERO NODE UNLOCKED, IENS RESERVED=100000
3111106.025823
INSERTION ENDS AT 3111106.02583
ELAPSED TIME: 7 SECONDS
APPROXIMATELY 14285 NODES PER SECOND
GTM>W ^C0X(101,0)
C0X TRIPLE^172.101I^658640^658640
GTM>W $O(^C0X(101,"AA"),-1)
658639
GTM>D EN^C0XTEST
INSERTION STARTS AT 3111106.025932
USING BULKLOAD
LOCKING ZERO NODE
ZERO NODE UNLOCKED, IENS RESERVED=100000
3111106.025932
INSERTION ENDS AT 3111106.025936
ELAPSED TIME: 4 SECONDS
APPROXIMATELY 25000 NODES PER SECOND
GTM>W ^C0X(101,0)
C0X TRIPLE^172.101I^758641^758641
GTM>W $O(^C0X(101,"AA"),-1)
758640
GTM>D EN^C0XTEST
INSERTION STARTS AT 3111106.030006
USING BULKLOAD
LOCKING ZERO NODE
ZERO NODE UNLOCKED, IENS RESERVED=100000
3111106.030006
INSERTION ENDS AT 3111106.030015
ELAPSED TIME: 9 SECONDS
APPROXIMATELY 11111 NODES PER SECOND
GTM>W ^C0X(101,0)
C0X TRIPLE^172.101I^858642^858642
GTM>W $O(^C0X(101,"AA"),-1)
858641