UK newspaper and magazine publishers lost almost £170m in digital revenue last year as technology designed to stop advertisements from appearing next to hard-hitting content, such as shootings and terrorism, also inadvertently blocked them from appearing in some of the most popular stories of the year.

Publishers found many articles related to some of the most well read and therefore commercially valuable stories of the year – on topics ranging from the Rugby World Cup to Game of Thrones – shorn of advertising.

When advertisers run digital campaigns they use keyword blacklists – stocked with trigger words such as “attack”, “death” and “sex” – that automatically stop ads running in potentially problematic stories that feature them.

While publishers have traditionally encouraged the use of such brand reputation protection measures in the digital age, blacklists are ballooning in some cases to as many as 3,000 or 4,000 words, blocking ads from many different stories.

The most searched phrase on Google last year was Rugby World Cup, but words including ‘attack’ meant many articles fell foul of ad-blocking. Photograph: Shaun Botterill/Getty Images

“An unintended consequence is that perversely some of the best and most trusted content is being blocked inadvertently,” said James Wildman, chief executive of Hearst UK, the publisher of magazines including Cosmopolitan and Elle. “If we write about the Duke and Duchess of Sussex [Harry and Meghan], chances are most software would see the word ‘sex’ and assume it is inappropriate content to appear alongside and block it. We are getting blocked for keywords like ‘shoot’. We write about photoshoots constantly.”

Newspaper and magazine publishers in the UK, US, Japan and Australia missed out on $3.2bn (£2.46bn) in digital revenue last year as “over-blocking” stopped ads appearing in articles with “safe or neutral content”, according to a report by the Merrick School of Business at the University of Baltimore, commissioned by cybersecurity firm CHEQ.

The report estimates that UK publishers missed out on £167m in advertising last year due to keyword ad-blocking. Publishers have found that up to 60% of their “safe” – non-controversial or neutral – content is incorrectly falling foul of ad-blocking.

It estimates that in total UK publishers made £758m from digital advertising last year, meaning that about one in every £5 of ad revenue is being lost to blacklist blocking. National newspapers made £275m in digital advertising last year, regional titles £212m and magazine publishers £271m.

In the UK, the most-searched phrase on Google last year was the Rugby World Cup. However, according to the report many of the most-read articles fell foul of ad-blocking for the use of phrases such as “attack” and “injury”. Some articles on Avengers: Endgame, the highest-grossing film of all-time, also suffered as they contained “attack” and “kill”.

One senior executive at a UK media agency, which handles the advertising for numerous household-name brands, cited a case of being asked by a client to create a blacklist comprised of “any negative words”. Another agency executive knows of multiple advertisers who added the word “is” to their blacklists, wanting to avoid having ads appear around material on the Islamic State (Isis) militant group, virtually guaranteeing zero ads appearing on any news sites. Wildman says that the word Manchester is still on many blacklists three years after the bombing at an Ariana Grande concert at Manchester Arena.

Embattled publishers are struggling to cope with the dominance of Google and Facebook, which control about 70% of the £12bn UK digital ad market

“Advertising technology can be very, very blunt and isn’t sophisticated enough to discern between genuinely harmful content and acceptable content,” said Rob Lynam, a media executive who has worked in ad agencies and at the publisher of the Daily Mail. “There are good intentions but I don’t think anyone has really realised who it is punishing the most or to what extent.”