Resource Considerations #

This section is intended to provide assistance in estimating the required memory and the resulting indexing speed of SemSpect on the basis of empirical values.

The following is a statistical evaluation of RDF SemSpect on the indexing speed and the memory requirements. For this, we considered altogether 26 datasets with up to 352M triples. The benchmark has been conducted for different JVM memory allocations in order to roughly estimate the memory requirements to support a desired amount of triples. Moreover, it compares our two indexing methods, namely the “one pass” and “two pass” approaches. In particular, we generate the index (compressed dictionary and triples) in a single parsing iteration (namely “one pass”: faster, higher memory consumption) or in two separate parsing iterations (“two pass”: slower, but less memory consumption).

For the experiments, we used SemSpect v16.0.1 and conducted them on an Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz, 6 cores with 2 threads per core, 64 GB DDR3 @ 1334MHz.

Memory Requirements #

The following table should be read as a lookup table: Assuming the JVM is allocated with a certain amount of memory (JVM Memory (GB)), how many triples can you expect to be able to index (Max. Num. Triples) with RDF SemSpect? Please note that a comparison of memory consumption of “one pass” against “two pass” for a specific memory setting should be treated with caution, as the results often refer to a different number of datasets. Furthermore, this table also lists the initial memory allocation for loading an existing index into SemSpect for exploration. Since SemSpect uses caching for performance reasons the latter will increase over time up to the given allocation limit.

Indexing Throughput #

The indexing throughput metric that can act as a factor to estimate the initial generation time of an index for a given RDF dump file before it can be explored in SemSpect. Assuming our dataset has 10,000,000 triples and we are using the “one pass” approach, the average required time corresponds to 10,000,000 / 50,500 ≈ 180(s) = 3(m). As can be seen, the maximum indexing throughput is much higher (factor ~3) since the individual speed depends on the dataset and its inherent characteristics such as the depth of the class and property hierarchy or the number of object property assertions in connection with the reasoning mode of SemSpect. As soon as an index has been created and saved on disk, it only takes a fraction of the indexing time to load it into memory for all subsequent calls of SemSpect.