Skip to content

Elasticsearch

Formbird uses 'Elasticsearch' as the method of indexing the database. There are various components available to present and use existing data such as sc-related-document and sc-datatables. These components require a 'search' to find documents relevant to the component function. For example if you wanted to view all 'Contacts', you would use an 'sc-datatables' component and write a search for the component to query for 'Contact' documents.

Let's have a quick look at sc-datatables:

        {
            "componentName": "sc-datatables",
            "name": "allContacts",
            "label": "All Contacts",
            "filter": "{'query':{'bool':{'filter':[{'term':{'appTags':'contact'}},{'term':{'systemHeader.systemType':'document'}} ] }}}",
            "fullWidth":true,
            "gridColumns": [
                {
                    "displayName": "Call Description",
                    "field": "systemHeader.summaryName",
                    "href": "/#/form/{{{documentId}}}",
                    "type": "url",
                    "urlOpenIn": "newWindow",
                    "width": 3
                }
        }

We're interested in this line from the component:
"filter": "{'query':{'bool':{'filter':[{'term':{'appTags':'contact'}},{'term':{'systemHeader.systemType':'document'}} ] }}}",

the 'filter' key is where the query is placed to return relevant documents. We won't cover the complete elasticsearch syntax in this document as the syntax is extensive. The elasticsearch reference document is available here : Elasticsearch Reference

The most common type of query used in Formbird is the 'term' query - eg {'term':{'appTags':'contact'}}
The above query is telling the component to look for all documents where 'systemHeader.systemType' = 'document' AND 'appTags' = 'contact'.
If you created a document using the 'Create a template in 5 minutes', this filter would find that document.
Example:

{
    "documentId": "f9b54310-bad0-11e8-9c0c-e3ee36fd7cf7",
    "systemHeader": {
        "templateId": "53bd7810-bad0-11e8-9c0c-e3ee36fd7cf7",
        "systemType": "document",
        "createdWith": "53bd7810-bad0-11e8-9c0c-e3ee36fd7cf7",
        "keyIds": [],
        "serverUpdatedDate": "2018-09-17T23:26:08.644Z",
        "serverCreatedDate": "2018-09-17T23:26:08.644Z",
        "versionId": "0e58ec40-bad1-11e8-ad0c-7b1e920f059f",
        "excludeGeneralSearch": false,
        "currentVersion": true,
        "createdDate": "2018-09-17T23:26:08.644Z",
        "createdBy": "5435d78388968497030106d0",
        "summaryName": "Contact: John Smith 23452345"
    },
    "appTags": [
        "myAppName",
        "contact"
    ],
    "contactName": {
        "title": "Mr",
        "firstName": "John",
        "surname": "Smith"
    },
    "phone": "23452345",
    "email": "john.smith@email.com",
    "contactAddress": {
        "type": "FeatureCollection",
        "features": [
            {
                "type": "Feature",
                "properties": {
                    "type": "street",
                    "unitNo": "",
                    "streetNo": "6",
                    "street": "RYRIE STREET",
                    "suburb": "GEELONG",
                    "postcode": "3220",
                    "locationDescription": "",
                    "country": "Australia",
                    "iconImage": "",
                    "iconSize": "",
                    "state": "VIC"
                },
                "geometry": {
                    "type": "Point",
                    "coordinates": [
                        144.3531069,
                        -38.14763674
                    ]
                }
            }
        ]
    }
}

The key fields are:

"appTags": [
    "myAppName",
    "contact"
],

and 

"systemHeader.systemType":"document"

Let's dive into the elastic query a bit more. We'll describe the most common query type in Formbird.

  • You'll notice that the query is a JSON object, so it can be treated like straight JSON when coding.
  • In Formbird, search elements are surrounded with single quotes (not double quotes as in the elasticsearch documentation)
  • Query terms are separated with a colon.
  • Handlebars can be used
  • The query is enclosed in double quotes - surrounding the entire query.

Filter

The query starts with the 'query' statement eg : {'query':

  • The next term in the query is 'bool' - a Bool Query is a combination of query types. So building on the previous ... {'query':{'bool'
  • The next term in the query is 'filter' - a Filter Query is a non-scored 'AND' query. Filter contains an array of 'query terms'. Building on previous .... {'query':{'bool':{'filter':[
  • The next term in the query is 'term' - a Term Query is an exact match for all elements in the array eg: "{'query':{'bool':{'filter':[{'term':{'appTags':'contact'}},{'term':{'systemHeader.systemType':'document'}} ] }}}"

As mentioned the 'bool' query can be a combination of query types, The two other most common query types are 'must_not' and 'should'

Must Not

A 'must_not' query is the opposite of filter (except it's a scored result) Lets say as an example we wanted to exclude the contact document described above, we could alter the above query to:

 "{'query':{'bool':{'filter':[{'term':{'appTags':'contact'}},{'term':{'systemHeader.systemType':'document'}} ],'must_not':[{'term':{'email':'john.smith@email.com'}} ] }}}"

This is saying return all documents where 'appTags' = 'contact' AND 'systemHeader.systemType' = 'document' AND NOT 'email' = 'john.smith@email.com'. We could use any term which matches the data on the document, in this instance we used 'email'. But the email address may change, and then it wouldn't be excluded anymore.
If this is a concern for your use, then it's best to pick a field on the document that doesn't change which is always the documentId. We'll rewrite that query ... 

"{'query':{'bool':{'filter':[{'term':{'appTags':'contact'}},{'term':{'systemHeader.systemType':'document'}} ],'must_not':[{'term':{'documentId':'f9b54310-bad0-11e8-9c0c-e3ee36fd7cf7'}} ] }}}"

Elastic 'filter' and 'must_not' queries are both AND queries.

Should

A 'should' query is the equivalent of an 'OR' query

Let's say we had 50 contact documents and we only wanted to find 'john.smith@email.com' and 'sarah.jensen@email.com' - how would that query be written? 

"{'query':{'bool':{'filter':[{'term':{'appTags':'contact'}},{'term':{'systemHeader.systemType':'document'}} ],'should':[{'term':{'email':'john.smith@email.com'}},{'term':{'email':'sarah.jensen@email.com'}}],'minimum_should_match':1 }}}"

This query is saying return all documents where 'appTags' = 'contact' AND 'systemHeader.systemType' = 'document' AND ('email' = 'john.smith@email.com' OR 'email' = 'sarah.jensen@email.com') The 'minimum_should_match':1 element in the query means that at least 1 of the terms must be matched. Let's add another email into the should statement and change 'minimum_should_match' to 2: 

"{'query':{'bool':{'filter':[{'term':{'appTags':'contact'}},{'term':{'systemHeader.systemType':'document'}} ],'should':[{'term':{'email':'john.smith@email.com'}},{'term':{'email':'sarah.jensen@email.com'}},{'term':{'email':'joe.brown@email.com'}}],'minimum_should_match':2 }}}"

In this instance ANY 2 email addresses should match and the query would return.

Handlebars

There is often a requirement to have the query modified dynamically. This can be achieved by using 'handlebars' in the query. The context of the form contains both the account data and the document data. These can be referenced in the following way:

 "{'query':{'bool':{'filter':[{'term':{'email':'{{document.email}}' }},{'term':{'systemHeader.systemType':'document'}} ] }}}"

This query is saying return all documents where 'email' = 'email that the user entered into the 'email' field on the document' AND 'systemHeader.systemType' = 'document' You would use a query like this to set up a form where the user could search for emails from people entered on the form

To reference the account information in handlebars use:

 "{'query':{'bool':{'filter':[{'term':{'reportedBy.documentId':'{{account.documentId}}' }},{'term':{'systemHeader.systemType':'document'}} ] }}}"

If you had an 'sc-related' field named 'reportedBy' this would match the current users documentId with ANY reportedBy.documentId

Terms

Multiple terms can be assembled in an array and queried using a 'terms' query. This is an OR query over the array

 "{'query':{'bool':{'filter':[{'terms':{'widgetColour':['brown','blue','green','fushcia'] }},{'term':{'systemHeader.systemType':'document'}} ] }}}"

Elasticsearch and Arrays

One feature of elasticsearch is that it flattens arrays. In the Addressing JSON document, we discussed how an array has an index and that to reference a particular element in the array, you need to refer to the index (eg. reportedBy[0].documentId). Elasticsearch does not require the array element to be referenced. It treats all of the data in the array (in simplistic terms) as one big text field to search over. Hence this example: 

 "{'query':{'bool':{'filter':[{'term':{'reportedBy.documentId':'{{account.documentId}}' }},{'term':{'systemHeader.systemType':'document'}} ] }}}"

uses 'reportedBy.documentId'. It doesn't matter how many documentId's the array contains, elastic search will search them all.

Note this doesn't apply to handlebars - If the field to be searched for is an element of an array it must be specified eg - let's suppose that the account document had an array of documentIds : 

"{'query':{'bool':{'filter':[{'term':{'thisArray.documentId':'{{account.arrayName.0.documentId}}' }},{'term':{'systemHeader.systemType':'document'}} ] }}}"

We would use account.arrayName.0.documentId to reference the array on the account.รท