No! This is a common misconception. Declining a test cycle will actually give you a small ratings boost. This is because it is far better to decline a cycle which you know you will not participate in than to not answer the invite.
Accepting a cycle that you do not participate in will hurt your rating.