The rule in all writing is that common names don't get capitalized. House, for instance, is common, because it can refer to multiple dwellings. Thus, unless it's at the beginning of a sentence, it isn't capitalized. White House, on the other hand, is the proper name of a very specific house.
I suppose in the case of finance department you could argue that you're referring to a very specific finance department. In which case I would invoke the Terry Rule: capitalized words are harder to read and you don't need them here so skip it. That's a good rule, I think.
Grammar Girl offers a reason for over capitalization:
One mistake business writers often make is capitalizing words simply for emphasis or to augment their importance. Such errant capitalization happens frequently in press releases and other promotional materials. Hyperbole is no stranger in that realm. Nevertheless, it does not make your pork rinds crunchier and tastier if you capitalize the words “Pork” and “Rinds.” Murray Munn commented on the “Pork Rind” kind of capitalization on the Grammar Girl Facebook page. He calls them “pride capitals” and speculates that “What we admire, we capitalize.” For example, he says he often sees librarians write “library” with a capital L.What about titles of people? The rule is that if the title comes before the name it's capitalized. If it comes after it's not.
When in doubt, or whenever you encounter someone with a lengthy official job title, give the person’s name first, then follow with the title, lowercase [for example]: Bartholomew Z. Bartholomew, 2nd assistant vice president for sales, northeast region, for Amalgamated Malaria Inc. His name and the company name are uppercase, but the rest of the words, such as “assistant vice president for sales,” are lowercase.That's so you won't end up with "Second Assistant Vice President For Sales, Northeast Region, for Amalgamated Malaria Inc. Bartholomew Z. Bartholomew. Although if that really is his name I don't think he would mind.
Here's a rule: When in doubt about capitalization, don't.