I edited the code in my question above.
I changed:
elseif( is_category() )
to:
elseif( is_category() || is_single() )
And so now it works for both pages and posts within a certain category, but not the categories themselves (the index/archive/category pages).
