[Dev] RDF tax

Uche Ogbuji uche.ogbuji at fourthought.com
Sun Nov 17 09:29:04 PST 2002

> One lesson I learned while working on Mozilla is that it can be tricky 
> to make a scalable data source using RDF, even if the underlying format 
> (beneath the RDF representation) is scalable.  As an example, consider 
> a large mailbox with 50,000 messages.  If you force the UI to display 
> information through communication with your RDF layer, then you can end 
> up with a real challenge on your hands as far as avoiding walking too 
> much of the RDF data (especially when hierarchies are involved).  This 
> problem is compounded by aggregation, where even simple things like 
> counting how many items are in a subfolder become challenging (since 
> all data sources have to get involved).  Trying to do projection in a 
> message display becomes very difficult, especially with aggregation.

I want to point out that we did run into some problems on this in large-scale 
RDF projects, and we came up with some solutions, even in the case of scaling 
to millions of triples.

One is 4Suite scopes.  You can narrow a query to a scope, which usually uses 
the underlying driver to narrow down the search space accordingly.

ANother is specific optimizations we made to the implementation of the Versa 
RDF query language in 4Suite.  The 4Suite Versa optimizer does a lot to take 
advantage of the back-end facilities for scalable query.

All that having been said, these solutions do not cover all potential RDF 
scalability problems, even if we were considering use of 4Suite for the RDF 
engine.  I think one would have to look at the specifc Chandler needs during 
design and put in some analysis.  I have some experience in this area, and I'm 
happy to lend it.

Uche Ogbuji                                    Fourthought, Inc.
http://uche.ogbuji.net    http://4Suite.org    http://fourthought.com
Python&XML column: 2. Introducing PyXML - http://www.xml.com/pub/a/2002/09/25/p
The Past, Present and Future of Web Services 1 - http://www.webservices.org/ind
The Past, Present and Future of Web Services 2 - 'http://www.webservices.org/in
Serenity through markup - http://adtmag.com/article.asp?id=6807
Tip: Using generators for XML processing - http://www-106.ibm.com/developerwork

More information about the Dev mailing list