Big data, machine learning and AI used by hedge funds to deal with Twitter ‘firehose’
In an effort to beat benchmarks, investment companies sometimes say they are looking at the entire dataset of Twitter, known in the business as the “full firehose”. In actual fact, few people can manage the sheer scale and storage challenges that come with it, not to mention the costs. A hypothesis-driven attempt to do some of this manually is possible but challenging; for instance, you could start searching the social media stream using a hashtag approach. Peter Hafez, chief data scientist at big data analytics firm RavenPack, knows the market well and how tricky it is to process large volumes of noisy unstructured data. He recounted a story about a small hedge fund, which tried the hashtag approach on “gold”, hoping to create a gold sentiment indicator to trade the…
Link to Full Article: Big data, machine learning and AI used by hedge funds to deal with Twitter ‘firehose’
Sharing
