Twitter Heron and Dhalion

2017-07-01 00:00:00 +0000

Twitter Heron and Dhalion

Two recent papers have given more detail on Twitter’s distributed stream processing system Heron (the successor to Storm).

The first paper, Twitter Heron: Towards Extensible Streaming Engines , details how Heron has evolved into a modular architecture that makes it extremely flexible. Adrian Colyer gives a good summary of the paper on his blog The Morning Paper. The main takeaway for me is the ease with which Heron can be augmented with new features. As I read it I saw how I could adapt my modelling work into a module that could aid in auto-scaling Heron topologies. Of course, not long after this the second paper: Dhalion: Self-Regulating Stream Processing in Heron came out. Someone had already done it, but this is by no means a bad thing.

As before, Adrian Colyer gives a good summary of the paper. Dhalion is a framework for regulating and maintaining a running Heron cluster. The authors have implemented monitoring, diagnostic and remedial systems to identify and fix performance issues and have show impressive results.

However, I see one issue with Dhalion’s current approach to resolving performance problems. The system currently has no way to know if a proposed resolution is likely to succeed before it is deployed. Dhalion will implement a resolution and observe its results in order to assess if it is successful. Updating a Heron topology (the equivalent of rebalancing in Storm) has a latency cost and it takes some time for the topology’s performance to stabilise after an update. Only after the topology has stabilised is it clear if a resolution has been successful or not. If not, then Dhalion has to repeat the process, potentially incurring further latency costs.

If Dhalion had a way to model the effects of a proposed resolution then it could iterate to an effective solution much faster. The authors of Dhalion have already done the lions share of the work, providing monitoring code and deployment options. I am looking forward to them open-sourcing Dhalion so I can investigate how easy it would be to integrate a modelling system, like the one I am developing for Storm, into the resolution process.

Python North East - The Next Generation

2017-03-01 00:00:00 +0000

Python North East - The Next Generation

This month I officially take the reins of the Python North East user group. Myself and Scott Walton are taking over from Rowan Hargreaves and Kieran Darcy’s fine stewardship.

I’m looking forward to the opportunity to meet many Python people and encourage new members to try the best programming language there is!

We are in the process of refreshing the website and have set up a new chat channel on Gitter. If you want to get in touch the best way is either on Gitter or via our Twitter account @pythonnortheast.

Hacking Happiness

2017-01-18 00:00:00 +0000

Hacking Happiness

Yesterday I took part in a hack-a-thon hosted by the Digital Catapult and the Royal Society. It was based on using data analysis and technology to try and measure and improve happiness. It was a great event, with lots of teams from all different disciplines and backgrounds. I was part of a team from Newcastle University, as well as members from Brighton and Central St. Martins.

Our idea was to create an app that could find the happiest route between two points, avoiding high pollution areas and preferring green spaces. We built a (rather simple) model using crime statistics and air quality data and an app to push route information to a back end service containing the model which would rate several routes on their “happiness” levels.

Another member of our team created a prototype “Happiness Band” to use measures of heart rate and galvanic skin response to gauge a persons happiness as they travelled around the city. We intended to use this data to provide objective measures of stress along the routes we suggested, as well as provide heat maps of stressful areas to augment our happiness model.

The team members from Brighton and Central St. Martins created UI mock-ups and user stories to show how the final app would look and feel.

After several intense hours work:

We had our prototype and gave our presentation to the judges:

There were many innovative and much more functional applications than ours. But in the end the judges liked our combination of machine learning, smart devices and UI he most and declared us the winners!

It was a great experience and I look forward to more hack-a-thons in the future!