Enterprise Framework

Software Solutions in the Enterprise

ElasticSearch - return back only specific fields in source

How to get ElasticSearch to return only specific source fields.  FYI, this is called Source Filtering on elastic.co's website.

Example: 1

in the request payload, specify 

       _source: ["fieldname1", "fieldname2"]


Example:

Sample Data

POST /library/books/_bulk

{ "index": { "_id": 1 } }

{ "title": "The quick brown fox", "price": 5, "colors": ["red","green","blue"] }

{ "index": { "_id": 2 } }

{ "title": "The quick brown fox jumps over the lazy dog", "price": 6, "colors": ["blue","yellow"] }

{ "index": { "_id": 3 } }

{ "title": "The quick brown fox jumps over the quick dog", "price": 7, "colors":  ["green", "purple"] }

{ "index": { "_id": 4 } }

{ "title": "The quick brown fox jumps over the stupid dog", "price": 8, "colors": ["green","yellow"] }

{ "index": { "_id": 5 } }

{ "title": "The quick brown fox jumped and farted", "price": 9, "colors": ["yellow", "purple"] }


Search with _source - return back only title and price

GET /library/books/_search

{

    "_source": [ "title", "price" ]

}


Example: 2

Sample Data


POST /library/books/6
{ 
  "title": "The quick brown fox", 
  "price": 5, 
  "colors": ["red","green","blue"],
  "contacts": [
    {
      "id": 1,
      "name": "john doe"
    },
    {
      "id": 2,
      "name": "john doe"
    },
    {
      "id": 3,
      "name": "john doe"
    }
  ]
}


Search with _source Example

GET /_search

{

    "_source": {

        "includes": [ "title", "price" ],

        "excludes": [ "colors" ]

    },

    "query" : {

        "term" : { "user" : "kimchy" }

    }

}


Reference Documentation:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-source-filtering.html

GraphQL with ElasticSearch


Official Elastic.co ElasticSearch client for NodeJS, Browser client

$ npm install elasticsearch

Resources:


Get graphql-compose-elasticsearch

$ npm install graphql graphql-compose elasticsearch graphql-compose-elasticsearch --save

Mac Sierra : Install Yarn Errors

Yarn install issue on Mac

After trying to install Yarn from command line:

$ brew update

$ brew install yarn

Warning: Building gcc from source: 

The bottle needs the Xcode CLT to be installed. xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun Error: gcc cannot be built with any available compilers. Install GNU's GCC brew install gcc

After researching, I need to install the latest xcode command line tools.

$ xcode-select --install

Then I was able to run Yarn install

$ brew install yarn

ElasticSearch 5.6 Template To Apply Dynamic Template Mapping or Properties to Any Index and Type

Problem:  Have an ElasticSearch Dynamic Template that will apply Templates and/or new properties to any new Index or Type created.  

Anytime an Index is created with a new type it should

  • Have a default DynamicTemplate
  • Add the properties "timestamp" and "creation_date". 

First use the following resources:

https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/default-mapping.html

That only gets you part way.  

Solution:

This was not easy to find, But what you will need to is set (Examples below):

  • "index_patterns": ["*"]  /* this will match all indexes created */
  • "mappings"."{type}".  This should be set to "_default_"

Note: this does not effect existing Templates.

PUT _template/template_1

{ 
    "index_patterns": ["*"],
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
        "_default_": {
            "_source": {
                "enabled": true
            },
            "dynamic_templates": [
                {
                    "id_integers": {
                        "match": "*_id",
                        "match_mapping_type": "long",
                        "mapping": {
                            "type": "long"
                        }
                    }
                },
                {
                    "date_fields": {
                        "match": "*_date",
                        "match_mapping_type": "date",
                        "mapping": {
                            "type": "date"
                        }
                    }
                },
                {
                    "json_fields": {
                        "match": "*_json",
                        "match_mapping_type": "*",
                        "mapping": {
                            "type": "text"
                        }
                    }
                },
                {
                    "int_template": {
                        "match": "_*",
                        "match_mapping_type": "string",
                        "mapping": {
                            "type": "keyword"
                        }
                    }
                }                
            ],
            "properties": {
                "creation_date": {
                    "type": "date",
                    "format": "yyyy-MM-dd"
                },              
                "timestamp": {
                    "type": "date",
                    "format": "epoch_millis"
                }
            }
        }
    }
}

Test:  POST /school/student/1

{ 
  "title": "The quick brown fox", 
  "price": 5, 
  "colors": ["red","green","blue"],
  "do_something":"hello world"
}

Validation:  POST /school/student/_mapping

{
  "school": {
    "mappings": {
      "student": {
        "dynamic_templates": [
          {
            "id_integers": {
              "match": "*_id",
              "match_mapping_type": "long",
              "mapping": {
                "type": "long"
              }
            }
          },
          {
            "date_fields": {
              "match": "*_date",
              "match_mapping_type": "date",
              "mapping": {
                "type": "date"
              }
            }
          },
          {
            "json_fields": {
              "match": "*_json",
              "mapping": {
                "type": "text"
              }
            }
          },
          {
            "int_template": {
              "match": "_*",
              "match_mapping_type": "string",
              "mapping": {
                "type": "keyword"
              }
            }
          }
        ],
        "properties": {
          "creation_date": {
            "type": "date",
            "format": "yyyy-MM-dd"
          },
          "timestamp": {
            "type": "date",
            "format": "epoch_millis"
          }
        }
      }
    }
  }
}

Mac Sierra - Amazon AWS CLI Easy Installer

http://docs.aws.amazon.com/cli/latest/userguide/awscli-install-bundle.html

    Switch to the download folder

    $ cd ~/Downloads

    Download the AWS CLI Bundled Installer.

    $ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"

    Unzip the package.

    $ unzip awscli-bundle.zip

    Note

    If you don't have unzip, use your Linux distribution's built in package manager to install it.

    Run the install executable.

    $ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

    Note

    By default, the install script runs under the system default version of Python. If you have installed an alternative version of Python and want to use that to install the AWS CLI, run the install script with that version by absolute path to the Python executable. For example:

    $ sudo /usr/local/bin/python2.7 awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws



Node.js - HTTP with Error, Done, Data, End, Timeout example


var https = require('https');
var querystring = require('querystring');
var postData = { data : "something" };
var options = {
hostname: 'www.yourdomain.com',
path: '/api/book',
method: 'POST',
headers: {
"Content-Type": "application/json",
"Content-Length": postData.length
}
};

var req = https.request(options, function (res) {
res.setEncoding('utf8');

res.on('data', function (chunk) {
console.log("[RESPONSE DATA]", chunk);
});

res.on('end', function (data) {

if (res.statusCode >= 200 && res.statusCode < 300) {
console.log("[SUCCESS]");
} else {
console.log("[HTTP RESPONSE STATUS CODE ERROR]", res.statusCode);
console.log("[HTTP RESPONSE STATUS MESSAGE ERROR]", res.statusMessage);
}
});
});
req.setNoDelay(true);
req.setTimeout(28000, function(e) {
console.log("Timeout Fired at 29000 ms");
});

req.on('error', function (e) {
console.log("[FORWARD ERROR]", e);

if (e.statusCode === "ECONNRESET") {
console.log("Timeout received");
}
});

req.write(postData);
req.end();

Lambda Node.js Callback

http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html


Using the Callback Parameter

The Node.js runtimes v4.3 and v6.10 support the optional callback parameter. You can use it to explicitly return information back to the caller. The general syntax is:

callback(Error error, Object result);



Where:

  • error – is an optional parameter that you can use to provide results of the failed Lambda function execution. When a Lambda function succeeds, you can pass null as the first parameter.

  • result – is an optional parameter that you can use to provide the result of a successful function execution. The result provided must be JSON.stringify compatible. If an error is provided, this parameter is ignored.

Note

Using the callback parameter is optional. If you don't use the optional callback parameter, the behavior is same as if you called the callback() without any parameters. You can specify the callback in your code to return information to the caller.

AWS Gateway Resource Expression Format

http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-control-access-using-iam-policies-to-invoke-api.html

The API-executing Resource expression has the following general format:

where:

  • region is the AWS region (such as us-east-1 or * for all AWS regions) that corresponds to the deployed API for the method.

  • account-id is the 12-digit AWS account Id of the REST API owner.

  • api-id is the identifier API Gateway has assigned to the API for the method. (* can be used for all APIs, regardless of the API's identifier.)

  • stage-name is the name of the stage associated with the method (* can be used for all stages, regardless of the stage's name.)

  • HTTP-VERB is the HTTP verb for the method. It can be one of the following: GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS.

  • resource-path-specifier is the path to the desired method. (* can be used for all paths).

Some example resource expressions include:

  • arn:aws:execute-api:*:*:* for any resource path in any stage, for any API in any AWS region. (This is equivalent to *).

  • arn:aws:execute-api:us-east-1:*:* for any resource path in any stage, for any API in the AWS region of us-east-1.

  • arn:aws:execute-api:us-east-1:*:api-id/* for any resource path in any stage, for the API with the identifier of api-id in the AWS region of us-east-1.

  • arn:aws:execute-api:us-east-1:*:api-id/test/* for resource path in the stage of test, for the API with the identifier of api-id in the AWS region of us-east-1.

  • arn:aws:execute-api:us-east-1:*:api-id/test/*/mydemoresource/* for any resource path along the path of mydemoresource, for any HTTP method in the stage of test, for the API with the identifier of api-id in the AWS region of us-east-1.

  • arn:aws:execute-api:us-east-1:*:api-id/test/GET/mydemoresource/* for GET methods under any resource path along the path of mydemoresource, in the stage of test, for the API with the identifier of api-id in the AWS region of us-east-1.