Sources of Machine Readable News

External question — Quantitative Finance Stack Exchange Author: gamerx Original post: https://quant.stackexchange.com/questions/4410 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I'm starting on a project that involves correlating and forecasting Forex time series to news releases. I'm aware of sources such as Thomson Reuter's machine readable news and Dow Jone's Newswire services. However, they both require expensive subscriptions. Does anyone know of any similar but free equivalent that I can use to develop a proof of concept? I think the news releases do not have to be strictly real time, and some delay is fine as well but the timestamp is essential. I think shouldn't have limited my question to just machine readable news which are preprocessed. What I'm interested in is more along the lines of Ransquawk's live headlines service which provides text-based news headlines updates. The text data aren't in a "machine readable" but that's fine as I can apply some simple NLP techniques on my side to extract information. The main issues is how to tap into these live feeds since most of them don't provide an API. And since what I'm working on is more a proof of concept, I think an archive timestamped news article can do as well.

External answer — Quantitative Finance Stack Exchange Author: Thomas Maloney Original post: https://quant.stackexchange.com/a/4412 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. News is not free, and hence you won't find a company offering machine readable news services for free. My best suggestion is to ask a machine readable news company for a day's worth of historical data. Even that might not work, however, as they won't waste their time if they don't think you're going to buy their service.

External answer — Quantitative Finance Stack Exchange Author: Matt Wolf Original post: https://quant.stackexchange.com/a/4414 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. Concur with Thomas for most part, though I would recommend you to sign up for a trial with Dow Jones Newswire. I like the API and app that Newsware ( http://www.newsware.com/ ) makes available. It is not suitable for hft but I use it in order to stay informed and look up often used mnemonics. I think they have a pretty capable API and I remember they offer trials. You may peruse it in order to calculate historical reactions to certain economic releases, however I am willing to bet you won't find ways to successfully (meaning, sustainable) forecast future price moves based on past news releases. I believe the only way to monetize price reaction to news and economic releases is through hft and employing a host of highly skilled linguists, programmers and economists. To my knowledge all software that is capable of analyzing text and deriving trading signals from news releases is stricly proprietary for the very reason that it costs a fortune to develop something that has an edge. Not exactly the playground for people on the retail side, and not even the playground for most investment banks, simply for lack of very specialized resources that are needed to be successful in this particular endeavor.

External answer — Quantitative Finance Stack Exchange Author: analystic Original post: https://quant.stackexchange.com/a/4428 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. It would be relatively trivial to implement a web scraper for any website you were interested in gathering news from - see Beautiful Soup for Python. This would allow you to gather and analyse news data from multiple sources in a way that may be more robust than relying on a single service. For example, you could screen scrape a certain website for the news headlines it's reporting, and then use a variety of statistical techniques to cluster and analyse the stories. I'm not sure if this would fit with the terms of use of certain websites, so not going to comment on that.

External answer — Quantitative Finance Stack Exchange Author: Jono Newton Original post: https://quant.stackexchange.com/a/4437 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. If you are an academic interested in this field I would suggest contacting Sirca. Thomson Reuters is active with academics through their partnership with Sirca in Australia (www.sirca.org.au). Sirca has other machine readable text products available.

External answer — Quantitative Finance Stack Exchange Author: Datageek Original post: https://quant.stackexchange.com/a/10646 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. Cost-wise the following sources look promising. While not strictly finance related, you can filter the input news down to the categories applicable to your needs: http://datasift.com/ http://www.dataminr.com/

External answer — Quantitative Finance Stack Exchange Author: Brian Risk Original post: https://quant.stackexchange.com/a/15039 License: CC BY-SA 3.0 — https://creativecommons.org/licenses/by-sa/3.0/ Adaptation: HTML converted to plain text; contact email addresses removed. Free sources of machine-readable news abound in the form of RSS or Atom XML feeds. These formats are so popular that many of your favorite programming languages already have libraries that makes parsing them a plug-n-play task. Here are a couple of big time feeds to get you started: http://feeds.bbci.co.uk/news/rss.xml http://rss.msnbc.msn.com/id/3032091/device/rss/rss.xml http://rss.cnn.com/rss/cnn_topstories.rss Feeds contain handy information such as title, time posted and often some summary of the content (first paragraph, first n characters, full content, etc.) Feel free to contact me if you need further help with parsing. I recently put together an aggregator of over 200 feeds for data mining, quants, stats and open data, so this is still fresh on my mind: Quandl.io

External answer — Quantitative Finance Stack Exchange Author: Jay Original post: https://quant.stackexchange.com/a/43416 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I ended up developing my own financial news API (real-time and historical) covering All newswires and press releases of all US listed companies (PR Newswire, Globenewswire, BusinessWire, etc) Journals (Wall Street Journal, Bloomberg, Reuters, etc.) All SEC filings (10Q/10K, 8K, 4, etc) + SEC press releases (e.g. SEC charges a company with fraud) clinicaltrials.gov FDA approvals, new drug applications, FDA press releases (fda.gov) US government contract awards (sam.gov) Patent applications, approvals and appeals (uspto.gov) US Department of Defense contract awards (e.g. $50M to Boeing) FCC filings (e.g. Amazon registering new ground stations for satellites) The indexation algortihm indexes new articles within 210 milliseconds (median) after publication and identifies any US company mentioned in the article in order to tag the article with the corresponding trading symbols. API example response: "articles": [ { "symbols": ["AAPL"], "title": "Apple to extend some free subscription to video-streaming service through February", "description": "Apple Inc said on Thursday that it would extend free Apple TV+ subscriptions, which were set to end within the next three months, through February.", "url": "https://newsfilter.io/articles/apple-to-extend-some-free-subscription-to-video-streaming-service-through-february-08e1183fa9faa738dc66ae6d2c7d4f3e", "publishedAt": "2020-06-08T15:50:02.473Z", "id": "25a5e721982ce5b75f340c2d30bc7bc3", "industries": [ "Software - Infrastructure", "Household & Personal Products" ], "sectors": ["Technology", "Consumer Defensive"], "source": { "id": "reuters", "name": "Reuters" } }, { "symbols": ["C"], "title": "Citigroup Sees Illinois Bonds Already Pricing In Worst Outcome", "description": "When it comes to Illinois bonds, Citigroup Inc. says the worst-case-scenario has already been priced in.", "url": "https://newsfilter.io/articles/citigroup-sees-illinois-bonds-already-pricing-in-worst-outcome-65c8a53206ed9f030bd4cb18552bdaf0", "imageUrl": "https://assets.bwbx.io/images/users/iqjWHBFdfxIU/i4xxGH1AndIg/v2/-1x-1.png", "publishedAt": "2020-06-08T17:30:17.263Z", "id": "65c8a53206ed9f030bd4cb18552bdaf0", "industries": ["Banks - Global"], "sectors": ["Financial Services"], "source": { "id": "bloomberg", "name": "Bloomberg" } }, { "symbols": ["TSLA"], "title": "Tesla Hails Air-Quality Effects of Shutdowns Elon Musk Called Fascist", "description": "Tesla Inc. is proffering a decidedly different take on the coronavirus-related shutdowns Elon Musk called “fascist” and sued over weeks ago by suggesting they may bolster the case for banning internal-combustion engine cars.", "url": "https://newsfilter.io/articles/tesla-hails-air-quality-effects-of-shutdowns-elon-musk-called-fascist-15524708278c27207c1fb617acc598c6", "imageUrl": "https://assets.bwbx.io/images/users/iqjWHBFdfxIU/i3CTYqmBlg6o/v1/-1x-1.jpg", "publishedAt": "2020-06-08T17:29:57.367Z", "id": "15524708278c27207c1fb617acc598c6", "industries": ["Auto Manufacturers"], "sectors": ["Consumer Cyclical"], "source": { "id": "bloomberg", "name": "Bloomberg" } } ] Documentation : https://developers.newsfilter.io/ Previously, I used https://newsapi.org/ . However, newsapi doesn't provide symbol tickers and doesn't include many sources relevant to investing/trading. Overview: free access to over 30,000 news sources world wide (US, Germany, India, Japan, etc.) RESTful API returning JSON excellent API documentation no throttling Example: Top Headlines Request: https://newsapi.org/v2/top-headlines?country=us&apiKey=YOUR_API_KEY Response: { status: 'ok', totalResults: 36, articles: [ { source: { id: 'cnbc', name: 'CNBC' }, author: 'Lauren Thomas', title: 'Target reports holiday same-store sales growth of 5.7%, maintains 2018 outlook - CNBC', description: "Target said its same-store sales climbed 5.7 percent during this past holiday season, topping growth of 3.4 percent a year ago and surpassing some analysts' expectations.", url: 'https://www.cnbc.com/2019/01/09/target-reports-2018-holiday-sales.html', urlToImage: 'https://fm.cnbc.com/applications/cnbc.com/resources/img/editorial/2017/12/04/104877704-RTX3JT8Z-2-shopping-target.1910x1000.jpg', publishedAt: '2019-01-10T11:30:10Z', content: 'Target saw a surge of shoppers head to its stores and website this past holiday season, a sign that its investments in store remodels and delivery services are paying off, and an early sign that consumers across the U.S. spent more on gifts this year. div &gt… [+3929 chars]' }, { source: { id: 'bloomberg', name: 'Bloomberg' }, author: null, title: 'Ford to Cut Thousands of Jobs in Europe, Eyes Plant Closures - Bloomberg', description: null, url: 'https://www.bloomberg.com/news/articles/2019-01-10/ford-to-cut-thousands-of-jobs-in-europe-eyes-plant-closures', urlToImage: null, publishedAt: '2019-01-10T10:00:00Z', content: "To continue, please click the box below to let us know you're not a robot." }, { source: { id: 'al-jazeera-english', name: 'Al Jazeera English' }, author: 'Al Jazeera', title: 'Turkey says will launch Syria attack if US delays troop pullout - Aljazeera.com', description: "Turkey and US remain at loggerheads over the future of Syrian Kurdish forces after Trump's decision to pull out troops.", url: 'https://www.aljazeera.com/news/2019/01/turkey-launch-syria-attack-delays-troop-pullout-190110092123874.html', urlToImage: 'https://www.aljazeera.com/mritems/Images/2018/12/16/fcef9a3bf7424348b7fe95c865939579_18.jpg', publishedAt: '2019-01-10T09:55:00Z', content: 'Turkey will go ahead with an offensive against Syrian Kurdish fighters in Syria if the United States delays the withdrawal of its troops from the war-torn country, Turkish Foreign Minister Mevlut Cavusoglu has said.' } // ... and many more ] };

External answer — Quantitative Finance Stack Exchange Author: Carrie Farram Original post: https://quant.stackexchange.com/a/44551 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. We have >20k global news sources in one API - multiple formats/delivery options. Everything is real time and up to micro for cross connections if you need algo-speed. We have the largest collection of global regulatory/disclosure wires in one pipe, plus all the leading heavy-weight financial newswires (rumors, broker ratings etc). All tagged with multiple time stamps, company identifiers, subjects, impact codes and more. We have sizable archives too for back-testing. Reach out to me for a free test feed and pricing. https://www.linkedin.com/in/carrie-farram-b443027/