Search engines will evaluate the links that a document contains. A related link on a web page is valuable content in and of itself, and is treated as such be search engines.
However, links to totally irrelevant or spam content can potentially hurt the rankings of a page. 
Linking to a “bad neighbourhood” of spam sites or even lots of irrelevant sites can hurt a site’s rankings.

The point is if you have an exchange links page, always go back to it and verify that the sites your a linking to and from are still relevant to your content, and they are still live.