We study the possibility for a unitary theory of partially-massless (PM) spin-two field interacting with Gravity in arbitrary dimensions. We show that parity invariant interactions respecting general covariance lead to a reconstruction of Conformal Gravity in even dimensions. In this case the unitarity is sacrificed. By relaxing the parity invariance, we find a possibility of a unitary theory in four dimensions, but the parity-odd cubic vertex cannot be written in usual metric variables. We comment on possible approaches that may allow for the formulation of this theory. Finally, by relaxing the general covariance, we show that a non-minimal coupling between massless and PM spin-two fields may lead to an alternative possibility of a unitary theory, that necessarily involves mixed-symmetry fields.