How to add Elasticsearch to your Sitecore – Part II

Continuing with my article on how to add Elasticsearch to your Sitecore, today I’ll be demonstrating how to index documents using NEST,  a .NET client of Elasticsearch. Next, I demonstrate how to add two buttons to the Developer tab in Sitecore that triggers an Index re-creation and an indexation of the content tree.

If you’ve missed the first part of this article, please read How to add Elasticsearch to your Sitecore – Part I.

I’ve uploaded the code that I created in this article to JANeto87/blog/tree/master/sitecore/elasticsearch/sitecore-elasticsearch.

NEST and Elasticsearch

Before adding any code to Sitecore, we’ll explore some basic usages of Elasticsearch.

There is a .NET client of Elasticsearch named NEST. It is the official high-level .NET client of Elasticsearch.

NEST has the advantage of having mapped all the request and response objects, comes with a strongly typed query DSL that maps 1 to 1 with the Elasticsearch query DSL, and takes advantage of specific .NET features such as covariant results. NEST internally uses, and still exposes, the low level Elasticsearch.Net client.

Example:

var elasticClient = new ElasticClient(new ConnectionSettings(new Uri("http://elastic.local:9200")));

var response = await client.GetAsync
<Article>(1, i =&gt; i.Index(indexName));

Let’s play a little bit with NEST.

Create a new Console Application project named “ElasticSearchDemo”.

New Console Application ElasticsearchDemo
Figure 1: Creating a new Console Application

Next, we’ll install the NEST NuGet package. You can use either the Nuget Package Manager (Tools > Nuget Package Manager > Manage Nuget Packages for Solution…) or install it through Package Manager Console (Tools > Nuget Package Manager > Package Manager Console):

PM> Install-Package NEST
Install-Package NEST
Figure 2: NEST NuGet package

Now, open the file Program.cs and add the following code:

namespace ElasticSearchDemo
{

    using Nest;

    class Program
    {
        static void Main(string[] args)
        {

            // Connecting to Elasticsearch
            var node = new Uri("http://elastic.local:9200");
            var settings = new ConnectionSettings(node);
            var client = new ElasticClient(settings);

        }
    }
}

To connect to Elasticsearch running locally at http://localhost:9200 you just need to instantiate a new instance of the Elastic client: var client = new ElasticClient().

In real scenarios, you need to pass additional configuration options to the client such as the address of Elasticsearch that is running on a remote machine. This is where ConnectionSettings come in.

var node = new Uri("http://elastic.local:9200");
var settings = new ConnectionSettings(node);
var client = new ElasticClient(settings);

Next, we’ll create a Plain Old CLR Object (POCO) named “Article” to represent the data we want to be indexed.

  1. In Solution Explorer, right-click the project name and click Add, and then click Add Class.
  2. The Add New Item dialog box appears with the C# Class icon already selected.
  3. In the dialog box, enter the name “Article” in the Name field and click Add. The new class is added to your project.
  4. Add the code below to the “Article” class.
public class Article
{
    public int Id { get; set; }
    public string Body { get; set; }
    public string Title { get; set; }
    public string Author { get; set; }
    public string Url { get; set; }
    public DateTime DatePublished { get; set; }
    public DateTime DateCreated { get; set; }
    public DateTime DateModified { get; set; }
}

Finally, we’ll create an index named “articles-index” and index a single instance of the Article class.

namespace ElasticSearchDemo
{
    using System;
    using Nest;

    class Program
    {
        static void Main(string[] args)
        {

            // Connecting to Elasticsearch
            var node = new Uri("http://elastic.local:9200");
            var settings = new ConnectionSettings(node);
            var client = new ElasticClient(settings);

            // Creating an index
            string indexName = "articles-index";
            var createResponse = client.CreateIndex(indexName);
            System.Console.WriteLine("The index {0} - has been created? {1}", indexName, createResponse.Acknowledged);

            // Indexing a document
            var article = new Article
            {
                Id = 1,
                Author = "kitty",
                Title = "Lorem ipsum dolor sit amet",
                Body = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed aliquam feugiat est ut condimentum.",
                DateCreated = new DateTime(2016, 12, 01),
                DateModified = new DateTime(2016, 12, 15),
                DatePublished = new DateTime(2016, 12, 20),
                Url = "https://goo.gl/tvcMLp"
            };

            var indexResponse = client.Index(article, x => x.Index(indexName));

        }
    }
}

In the code above we created an index named “articles-index” by calling the method client.CreateIndex(indexName). Next, we checked if the index was created successfully by checking the property createResponse.Acknowledged. Then, we created a single instance of the Article class and indexed it by calling the method client.Index(article, x => x.Index(indexName)).

Now, open http://elastic.local:9200/_plugin/kopf in a browser to see your index created. I am assuming you have followed through the first part of this article and installed the kopf plugin.

Kopf showing the index articles-index
Figure 3: Visualizing the index created

If you have not installed the kopf plugin, then just open http://elastic.local:9200/articles-index in a browser. You should see a JSON result like the following:

{
  "articles-index": {
    "aliases": {},
    "mappings": {
      "article": {
        "properties": {
          "author": {
            "type": "string"
          },
          "body": {
            "type": "string"
          },
          "dateCreated": {
            "type": "date",
            "format": "strict_date_optional_time||epoch_millis"
          },
          "dateModified": {
            "type": "date",
            "format": "strict_date_optional_time||epoch_millis"
          },
          "datePublished": {
            "type": "date",
            "format": "strict_date_optional_time||epoch_millis"
          },
          "id": {
            "type": "long"
          },
          "title": {
            "type": "string"
          },
          "url": {
            "type": "string"
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1495287853379",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "0RQbgs5YTm6VF0KoK9zvAQ",
        "version": {
          "created": "2030499"
        }
      }
    },
    "warmers": {}
  }
}

Now, how do we get a document? It is as simple as adding the code below:

 // Getting a document
var getResponse = client.Get
<Article>(1, i =&gt; i.Index(indexName));
Article articleDocument = getResponse.Source; // the original document

In the code above the getResponse.Source returns a typed Article object from the index.

And how do we use search to find an article that contains a given text inside of it?

var searchResponse = client.Search
<Article>(s =&gt;
            s.Index("articles-index").Size(10).Query(q =&gt;
            q.Match(m =&gt; m.Field(f =&gt;
            f.Author).Query("kitty"))));

var articles = searchResponse.Documents;

In the code above we are searching for all the articles that have “kitty” as the Author. On top of that, we are delimiting the size of the results by 10. Last, but not least, we are using the Match Query, which basically is a boolean query that looks for the term “kitty” inside the field Author and returns true if it is found. There are other types of queries such as Multi-Match, Fuzzy and others.

Integrating Elasticsearch with Sitecore

Now that we played with Elasticsearch, let’s add it to Sitecore.

For this article, I created a ElasticSearch module to add to a Habitat installation. Basically, you will need to add a new ASP.NET Web Application using the Empty template and the .NET Framework 4.6. Obviously, there are some additional steps to setup the Sitecore solution that won’t be covered here.

If you want to know how to install Habitat, please see https://github.com/Sitecore/Habitat/wiki/01-Getting-Started.

Don’t forget to install the NEST NuGet package:

PM> Install-Package NEST

In Sitecore, we’ll add two buttons to the Developer tab. The first, named “Re-Create Elastic Indexes”, will be responsible for re-creating the Elasticsearch indexes. The second, named “Re-Index Elastic Tree”, will be responsible for updating the Elasticsearch indexes for the selected item and its descendants in the Content Tree.

Sitecore Elasticsearch solution
Figure 4: Visual Studio solution

To add the mentioned buttons to the Developer tab, in the Core database, you should add them under /sitecore/content/Applications/Content Editor/Ribbons/Chunks/Indexing using the /sitecore/templates/System/Ribbon/Large Button template.

Re-Create Elastic Indexes button
Item name: RecreateElasticIndexes
Header: Re-Create Elastic Indexes
Icon: Office/32×32/cloud_flash.png
Click: demo:elastic:indexing:recreateindexes
Tooltip: Recreates all elastic indexes (removes all existing content, a refresh tree command will be required after to re-populate the indexes)

Re-Index Elastic Tree button
Item name: RefreshElasticTree
Header: Re-Index Elastic Tree
Icon: Office/32×32/cloud_refresh.png
Click: demo:elastic:indexing:refreshtree
Tooltip: Rebuild the elastic master and web index for this item and its descendants.

Here is how my Sitecore solution looks like after adding the definition item above.

Buttons to the Developer Tab
Figure 5: Elasticsearch button in Developer tab

First, create the NewsArticle model class with the following code:

using Nest;

public class NewsArticle
{
    [Text]
    public string Id { get; set; }

    public string Title { get; set; }
    public string Image { get; set; }
    public string Date { get; set; }
    public string Summary { get; set; }
    public string Body { get; set; }
}

The important thing about this code is the field Id which will help Elasticsearch to avoid duplicating your documents every time you reindex de tree.

Next, create the Templates class with the following code:

using Sitecore.Data;

public struct Templates
{
    public struct NewsArticle
    {
        public static readonly ID ID = new ID("{B69277AD-E917-4B9F-9136-A12E0A3E462F}");

        public struct Fields
        {
            public static readonly ID Title = new ID("{BD9ECD4A-C0B0-4233-A3CD-D995519AC87B}");
            public const string Title_FieldName = "NewsTitle";

            public static readonly ID Image = new ID("{3437EAAC-6EE8-460B-A33D-DA1F714B5A93}");

            public static readonly ID Date = new ID("{C464D2D7-3382-428A-BCDF-0963C60BA0E3}");

            public static readonly ID Summary = new ID("{9D08271A-1672-44DD-B7EF-0A6EC34FCBA7}");
            public const string Summary_FieldName = "NewsSummary";

            public static readonly ID Body = new ID("{801612C7-5E98-4E3C-80D2-A34D0EEBCBDA}");
            public const string Body_FieldName = "NewsBody";
        }
    }

}

Don’t forget to update the Template ID and Field IDs with the corresponding ones of your Sitecore instance.

Then, create the RefreshTree class and add the following code:

using Demo.Feature.ElasticSearch.Models;
using Nest;
using Sitecore;
using Sitecore.Configuration;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Diagnostics;
using Sitecore.Jobs;
using Sitecore.Shell.Applications.Dialogs.ProgressBoxes;
using Sitecore.Shell.Framework.Commands;
using System;
using System.Collections.Generic;
using System.Linq;

[Serializable]
public class RefreshTree : Command
{

    protected Handle JobHandle { get; set; }

    public override void Execute(CommandContext context)
    {
        Assert.ArgumentNotNull(context, "context");

        var item = context.Items[0];

        Assert.IsNotNull(item, "context item cannot be null");

        var progressBoxMethod = new ProgressBoxMethod(Refresh);

        ProgressBox.Execute(
            string.Format("{0} ({1})", "Re-Index Elastic Tree.", item.Paths.ContentPath),
            "Re-indexing the current item and its descendants in Elastic",
            progressBoxMethod,
            new object[] { item });
    }

    private void Refresh(object[] parameters)
    {

        JobHandle = Sitecore.Context.Job.Handle;

        if (parameters.Length != 1)
            return;

        var contextItem = parameters[0] as Item;

        if (contextItem == null)
            return;

        var job = JobManager.GetJob(JobHandle);

        var items = new List();
        if (contextItem.TemplateID == Templates.NewsArticle.ID)
            items.Add(contextItem);

        items.AddRange(contextItem.Axes.GetDescendants()
            .Where(s => s.TemplateID == Templates.NewsArticle.ID));

        List newsArticles = items.Select(news => new NewsArticle
        {
            Id = news.ID.ToString(),
            Body = news[Templates.NewsArticle.Fields.Body],
            Date = news[Templates.NewsArticle.Fields.Date],
            Image = news[Templates.NewsArticle.Fields.Image],
            Summary = news[Templates.NewsArticle.Fields.Summary],
            Title = news[Templates.NewsArticle.Fields.Title]
        }).ToList();

        if (job != null)
            job.Status.Messages.Add(string.Format("Indexing: {0} entries", newsArticles.Count));

        var response = IndexNews(newsArticles);

        if (response != null)
            job.Status.Messages.Add(string.Format("Indexing result: {0}", response.DebugInformation));

    }

    private IBulkResponse IndexNews(IEnumerable newsArticles)
    {

        if (newsArticles == null || !newsArticles.Any())
            return null;

        // Connecting to Elasticsearch
        string protocol = Settings.GetSetting("ElasticSearch.Protocol", "http");
        string host = Settings.GetSetting("ElasticSearch.Host", "elastic.local");
        string port = Settings.GetSetting("ElasticSearch.Port", "9200");

        var node = new Uri(string.Format("{0}://{1}:{2}", protocol, host, port));
        var settings = new Nest.ConnectionSettings(node);
        var client = new Nest.ElasticClient(settings);

        // Reindexing items
        var indexName = Settings.GetSetting("ElasticSearch.ArticlesIndex", "articles-index");

        var indexerResponse = client.IndexMany(newsArticles, indexName);

        return indexerResponse;
    }

}

In this code, I create a collection of Sitecore items that match the condition TemplateID == Templates.NewsArticle.ID, then I Invoke the method IndexNews that will connect to Elasticsearch and do a bulk index through the client.IndexMany(newsArticles, indexName) method.

Finally, create the ReCreateIndexes class with the following code:

using Sitecore.Diagnostics;
using Sitecore.Jobs;
using Sitecore.Shell.Applications.Dialogs.ProgressBoxes;
using Sitecore.Shell.Framework.Commands;
using System;

[Serializable]
public class ReCreateIndexes : Sitecore.Shell.Framework.Commands.Command
{

    protected Handle JobHandle { get; set; }

    public override void Execute(CommandContext context)
    {
        Assert.ArgumentNotNull(context, "context");

        var item = context.Items[0];

        Assert.IsNotNull(item, "context item cannot be null");

        var progressBoxMethod = new ProgressBoxMethod(Recreate);

        ProgressBox.Execute(
            "Recreate Elastic Indexes.",
            "Recreating all Elastic indexes.",
            progressBoxMethod,
            new object[] { item });

    }

    private void Recreate(params object[] parameters)
    {
        JobHandle = Context.Job.Handle;

        if (parameters.Length != 1)
            return;

        var item = parameters[0] as Item;

        if (item == null)
            return;

        var job = JobManager.GetJob(JobHandle);

        // Connecting to Elasticsearch
        string protocol = Settings.GetSetting("ElasticSearch.Protocol", "http");
        string host = Settings.GetSetting("ElasticSearch.Host", "elastic.local");
        string port = Settings.GetSetting("ElasticSearch.Port", "9200");

        var node = new Uri(string.Format("{0}://{1}:{2}", protocol, host, port));
        var settings = new Nest.ConnectionSettings(node);
        var client = new Nest.ElasticClient(settings);

        // Re-creating index
        var indexName = Settings.GetSetting("ElasticSearch.ArticlesIndex", "articles-index");

        DisplayStatusMessage(job, string.Format("Deleting '{0}' index", indexName));
        var deleteResponse = client.DeleteIndex(indexName);
        DisplayStatusMessage(job, string.Format("The index {0} - has been deleted? - {1}", indexName, deleteResponse.Acknowledged));

        DisplayStatusMessage(job, string.Format("Creating '{0}' index", indexName));
        var createResponse = client.CreateIndex(indexName);
        DisplayStatusMessage(job, string.Format("The index {0} - has been created? {1}", indexName, createResponse.Acknowledged));

    }

    private void DisplayStatusMessage(Job job, string message)
    {
        if (job != null)
            job.Status.Messages.Add(message);
    }
}

There is nothing new in the code above. We are just deleting and creating the index as we did in the Console Application.

Last, but not least, create the configuration file and add the following:

Don’t forget to publish the Visual Studio solution to your webroot folder.

Now, open you habitat instance and run the Re-create Elastic Indexes command in the Developer tab.

Next, select the item in the node /sitecore/content/Habitat/Home/Modules/Feature/News/News and run the Re-Index Elastic Tree command in the Developer tab.

Open http://elastic.local:9200/_plugin/kopf/#!/rest in a browser, and run the request “articles-index/_search”. You should see the Sitecore items indexed like the following.

articles index search kopf
Figure 6: Querying the indexed Sitecore items using kopf

In this article, I demonstrated one of the ways you can integrate your Sitecore with Elasticsearch. Obviously, that doesn`t show the full potential of Elasticsearch. I’ll keep writing more about some of the powerful things you can do with Elastic search. Wait for the next articles

References

  1. Elastic/Elasticsearch-Net“. GitHub. N.p., 2017. Web. 11 June 2017.
  2. Getting Started | Elasticsearch.Net And NEST: The .NET Clients [5.X] | Elastic“. Elastic.co. N.p., 2017. Web. 11 June 2017.
  3. Home – Schema.Org“. Schema.org. N.p., 2017. Web. 11 June 2017.
  4. Introduction | Elasticsearch.Net And NEST: The .NET Clients [1.X] | Elastic“. Elastic.co. N.p., 2017. Web. 11 June 2017.
  5. Match Query | Elasticsearch Reference [5.4] | Elastic“. Elastic.co. N.p., 2017. Web. 11 June 2017.
  6. Quick Start | Elasticsearch.Net And NEST: The .NET Clients [1.X] | Elastic“. Elastic.co. N.p., 2017. Web. 11 June 2017.

2 thoughts on “How to add Elasticsearch to your Sitecore – Part II

  1. The Sitecore 9 required the Solr but to implement Elastic search we need to setup Elastic search as mentioned at https://joaoneto.blog/2017/05/16/how-to-add-elasticsearch-to-your-sitecore-part-i/
    Is the above installation guide will be valid for SC9?
    Also, as per my understanding Elastic search will help to search content from different-2 source of data (like data from 3rd party systems , Sales Force and others..) and for this API’s should be available from other systems?

    Like

  2. Sitecore 9 requires Solr for the Sitecore client (/sitecore). In my post, I try to exemplify how to use Elastic search for the content search (e.g. a search bar in your site). I haven’t tested it on Sitecore 9, but it should work as it mostly uses Elastic assemblies.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s