Hotwire Tech Blog

Scribes from Hotwire Engineering

Problem:

Scoring Service[1] is a microservice used at Hotwire to determine the order in which hotels get displayed to a user corresponding to her searched destination. It uses machine-learning models developed by the Data Science team to compute scores of hotels based on its features. Hotwire Search Team has built a robust container- based infrastructure that allows Data Scientists to experiment with different sorting algorithms in order to provide the most relevant results to our users. Data plays a very important role when training new models. A sort model can only be as good as the data it is trained on. Thus, it is very important to be able to access real time application data and leverage it for online or offline learning and modeling. Scoring service generates a very high volume of log data that needs to be aggregated and persisted. Thus, we needed to develop or adopt a system that provides a scalable solution for handling large volume of data for not only Scoring Service, but also for other microservices in Hotwire architecture.

Solution:

Data Applications Team at Hotwire has built a simple and scalable solution to solve the log aggregation problem. Their system called Kivas is an API around Kafka. Kafka is a high throughput, low latency message broker that follows publish-subscribe messaging framework. The Search team at Hotwire decided to adopt Kivas and has built a system that will send its large requests and responses to Kafka. Kivas API provides an abstraction that prevents all microservices from having to build their own clients for Kafka. This enables scoring service to keep its interface clean and simple. Since log data is published to Kafka topics, it allows multiple consumers to independently listen to these topics and consume messages and write their own transformers depending on their use cases. In addition to having an API to publish data to a topic, Kivas also provides an extractor that consumes data from a topic and persist it in Amazon S3 buckets.

There are numerous benefits to adopting this approach of logging:

  1. Large volume of data is now logged in a scalable system.
  2. Data Science team can now independently access data from S3 storage.
  3. There is no engineering team’s involvement since scoring service does not have to be released in production in order to turn request/response-logging on/off. The previous approach of manually doing a configuration change to log data to host machine’s file systems was inconvenient and prone to error. The large volume of data would often fill up the disk space on hosts if logging remained on for a longer duration impacting performance and availability of Scoring Service. Thus, automating log data aggregation using Kivas has been a huge relief.
  4. Data Science can leverage real time, reliable application data to train and test the new sort models. Since the data quality of real time application data is better than ETL-transformed data, the resulting sort models are more likely to provide an accurate scoring mechanism for users.

Architecture:

pic

The above diagram represents the current architecture of Scoring Service. Scoring Service request sent from the client comprises of search criteria entered by the user, customer information, client device details and indexes for hotels that are available and priced for that search. Using these indexes, scoring service looks up static information about hotels using our Hotel Static Data Service API. The static hotel information consists of characteristics of hotels that do not change often, ex. amenities, lodging type, star ratings, recommendation scores, neighborhood details, etc. A full request is then constructed for the sort models using data obtained from scoring service client and Hotel Static Data Service API and the hotels are then scored based on the data in the request and the implementation of the sort models. Asynchronously, outside of the search path, this data log is sent to Kivas API which publishes that log to a Kafka topic. The extractor built within Kivas consumes the data from the topic and persists that data into S3 bucket. The data can then be retrieved and processed using Elastic Map Reduce (EMR) cluster.

Performance:

As a result of adopting Kivas/Kafka framework for log aggregation, there was no impact on the performance of Scoring Service since logs are published to Kafka asynchronously on a separate worker thread.

Next Steps:

  1. Leverage real time data for online learning and training of sort models
  2. Standardize the adoption of Kivas framework to aggregate logs for data analysis and modeling across other microservices within the Hotwire architecture.

Teams Involved:

Search, Data Science, Data Applications

[1] Introducing Microservice Architecture at Hotwire

Leave a Reply

Your email address will not be published. Required fields are marked *