Unstable string states in curved spacetime is not well-understood compared to those in the flat spacetime. We consider potentially tachyonic open string states in AdS_5xS^5 which should correspond to a determinant-like operator in N=4 super Yang-Mills by the AdS/CFT correspondence. Its conformal dimension is studied using integrability and perturbative methods in N=4 SYM. By proposing and solving the boundary thermodynamic Bethe ansatz (BTBA) equations, we find an indication that states of N=4 SYM turn into tachyonic at finite coupling where the total energy of the corresponding string becomes zero.