Here are some of the problems with pointer semantics.
If x is assigned to y, and then changes are made to the internal state of x, these will be reflected in changes having been made to y. The unwary programmer may be surprized by this.
If our language, such as Object Pascal, requires explicit memory management, and x is explicitly freed, what happens when the user tries to access the value of y? What happens if they try to free the value of y?
Some languages, such as C++, allow the programmer to change the meaning of the assignment arrow. In these languages the programmer can make assignment mean whatever they want.
If pointer semantics are uses, generally languages provide some way to make copies. However, Object Pascal and Java use pointer semantics, but provide only weak tools for making copies. Smalltalk and Objective-C are slightly better in this regard, as they also use pointer semantics, but do provide several built-in techniques for making copies of values.