Yes, if you can pic/scrape or otherwise remove the offending area, you "might" be ok as far as adhesion goes, but that's not the only problem. If you have any kind of "bulge" in that area, the abrasive will make harder contact with your project in that place. This will do a couple of bad things. First would be swirl marks and second, it will likely interfere with the orbit of the sander itself.
The pad needs to sit flat on the surface to work correctly. This is evidenced by the way you wore those spots in your pad. Minimal contact changes the orbit, more completely it changes the spinning part of the random orbit.
If you can get it removed enough to be a low spot, you may be able to extend its life a little, but IMHO you need a new one. Keep that one for "abusive situations" and swap it accordingly.
