Event logging with ElasticSearch

There is a lot of web analytics services available which have great products and you will most likely be well served using them. However if you are interested in building your own web analytics you can now use a little tool (elastic-event-js) we made to dispatch and query events on the browser to an ElasticSearch server through its REST API.

There's an official client library elasticsearch-js but our tool will manage for you a log of events and save them in bulk to the server from time to time. It will also save on unload of the window (when the user closes the browser tab) and make sure no events are lost. It will not send pre-flight requests and has no dependencies which makes it very small (3.13 KB minified).

We have a example on tracking the mouse movements and clicks to demonstrate how to use it. For this example we will be using logsene because it exposes an ElasticSearch API and has a free tier.

If you are building your own web analytics you shouldn't send this directly from the client-side for security reasons as we are doing in this example. You should instead use a proxy that adds in the required authentication with the elastisearch server, validates and deals with the authorization of requests. You may also mix-in other data from the server. There's a interesting article from elastic.co on using NGINX for this purpose.

For this example we will need to first configure ElasticEvent:

var elasticevent = new ElasticEvent({  
  // logsene server
  host: 'https://logsene-receiver.sematext.com',
  // This index from logsene shouldn't be public but for this example it will do
  // you can create your own
  index: 'your-index-here',
  // this will make sure that from 10 to 10 seconds it saves the
  // logged events, also there's a max of 256 events by default
  setupIntervalSend: true,
  // this will setup the listeners to the window unload event so that everything logged
  // will be saved
  setupBeforeUnload: true
});

We can extend the baseEvent method so that all our events will have a timestamp.

// extending base event
var base = elasticevent.baseEvent;  
elasticevent.baseEvent = function () {  
  var baseEvent = base.apply(this, arguments);
  baseEvent['@timestamp'] = new Date().toISOString();
  return baseEvent;
};

We should distinguish between different sessions, with a generated unique id:

elasticevent.identify({  
  sessionId: generateUniqueId()
});

Now we need to track events.

function track(type, event) {  
  elasticevent.track(type, {
    x: (event.clientX / window.innerWidth).toPrecision(8),
    y: (event.clientY / window.innerHeight).toPrecision(8)
  });
}
document.onclick = track.bind(null, 'click');  
document.onmousemove = track.bind(null, 'position');  

We are now tracking events, let's query them every two seconds and show the results. We can use the bodybuilder module to help us build the query object.

// keep the amount of results we displayed so we can query only for new results
var queryFrom = 0;  
setInterval(function() {  
  elasticevent.search(
    new Bodybuilder()
      // limit the query to the current sessionId
      .filter('term', 'sessionId.raw', elasticevent.traits.sessionId)
      .sort('@timestamp')
      // return 50 results
      .size(50)
      // pagination from the last results
      .from(queryFrom)
      .build('v2'),
      null,
    function(err, resp) {
      // if not new results do nothing
      if (!resp.hits.hits.length) return;

      queryFrom += resp.hits.hits.length;

      // display results
      console.log(resp);
    });
}, 2000);

And that's it. We have this example with drawing to canvas of the movements and clicks, which are then painted over with the results on the server. Feel free to play with it: Track Mouse Example and source code

comments powered by Disqus