How indexing works in SmartStore

Indexers handle buckets in SmartStore indexes differently from buckets in non-SmartStore indexes.

Bucket states and SmartStore

Indexers maintain buckets for non-SmartStore indexes in these states:

• Hot buckets
• Warm buckets
• Cold buckets

Indexers maintain buckets for SmartStore indexes in these states only:

• Hot buckets
• Warm buckets

The hot buckets of SmartStore indexes reside on local storage, just as with non-SmartStore indexes. Warm buckets reside on remote storage, although copies of those buckets might also reside temporarily in local storage.

The concept of cold buckets goes away, because the need to distinguish between warm and cold buckets no longer exists. With non-SmartStore indexes, the cold bucket state exists as a way to identify older buckets that can be safely moved to some type of cheaper storage, because buckets are typically searched less frequently as they age. But with SmartStore indexes, warm buckets are already on inexpensive storage, so there is no reason to move them to another type of storage as they age.

Buckets roll to frozen directly from warm.

Cold buckets can, in fact, exist in a SmartStore index, but only under limited circumstances. Specifically, if you migrate an index from non-SmartStore to SmartStore, any migrated cold buckets will use the existing cold path as their cache location, post-migration.

In all respects, cold buckets in SmartStore indexes are functionally equivalent to warm buckets. The cache manager manages the migrated cold buckets in the same way that it manages warm buckets. The only difference is that the cold buckets, when needed, will be fetched into the cold path location, rather than the home path location.

The indexing process

The indexing process is the same with SmartStore and non-SmartStore indexes.

The indexer indexes the incoming data and writes the data to hot buckets in local storage. In the case of an indexer cluster, the source peer streams the hot bucket data to target peers to fulfill the replication factor.

When buckets roll to warm, however, the SmartStore process differs from non-SmartStore.

Warm bucket handling

Starting from the point that a bucket rolls from hot to warm, the indexer handles SmartStore indexes differently from non-SmartStore indexes.

When a bucket in a SmartStore index rolls to warm, the bucket is copied to remote storage.

The rolled bucket does not immediately get removed from the indexer's local storage. Rather, it remains cached locally until it is evicted in response to the cache manager's eviction policy. Because searches tend to occur most frequently across recent data, this process helps to minimize the number of buckets that need to be retrieved from remote storage to fulfill a search request.

After the cache manager finally does remove the bucket from the indexer's local storage, the indexer still retains metadata information for that bucket in the index's .bucketManifest file. In addition, the indexer retains an empty directory for the bucket.

In the case of an indexer cluster, when a bucket rolls to warm, the source peer uploads the bucket to remote storage. The source peer continues to retain its bucket copy in local cache until, in due course, the cache manager evicts the copy.

After successfully uploading the bucket, the source peer sends messages to the bucket's target peers, notifying them that the bucket has been uploaded to remote storage. The target peers, like the source peer, continue to retain their bucket copies in local cache until, in due course, their cache managers evict the copies.

During the upload process, if the target peers do not hear from the source peer within five minutes, they query the remote storage to learn whether the bucket was uploaded. If it wasn't uploaded by the source peer, one of the target peers then uploads it.

When a bucket copy does get evicted from a peer's local cache, the peer retains metadata for the bucket, so that the cluster has enough copies of the bucket, in the form of its metadata, to match the replication factor.

In addition to retaining metadata information for the bucket, the source peer continues to retain the primary designation for the bucket. The peer with primary designation fetches the bucket from remote storage when the bucket is needed for a search.

Bucket freezing and SmartStore

See Configure data retention for SmartStore indexes.