Primarily, it's OK to say no, and it's OK for people to be unhappy with the answer - get over it. Your job isn't to make everyone happy or be the most loved employee, or the SE champion - it's to create market leading successful products. Also, recognize that saying yes when inappropriate can be the source of damage, difficulty, and failure.
Showing integrity is standing by your convictions and being truthful. Respect is gained by telling the truth and delivering, rather than constantly promising and not delivering. Sure, we may be uncomfortable with the no, but building a long term relationship is about honestly, reliability and mutual respect. You don't get that by misleading or downright lying.
However, before you say no, make sure the requester knows you have heard them - often people just want to ensure that you understand their pain (i.e. show empathy). A good way to do this is to listen, then reflect the request back to them e.g. "so what I heard you say was...". Once they hear the reflection they know that you have at least heard their concern/issue/pain. And it's OK to be sympathetic.
Next, understand the context of the request, how the requester encodes/decodes the world around them, and their motivation. If the requester is a sales person then likely they have a deal riding on this feature, a deal means quota, which translates to $$$$ in their pocket. So by saying no you are costing them money. In this case, you'll have to explain the decision in terms they understand e.g. "if we do this feature it isn't repeatable (so you can't sell it again), and we need to change our plans somewhere else. We may have to slow down the team implementing (insert cool new thing), and if we make that late then all your customers and prospects will get pushed back, so the next quarter will be tough".
What we are trying to do is get the requester to understand how and why we make decisions. You should have the vision, business plan and roadmap to show them why the product is on a path that makes them successful. Customize presentations and dialog around the requester e.g. engineers will likely want to see quantitative evidence why they should be working on the strategic direction rather than their pet project. If you can't find quantitative evidence, take them on a field trip to meet with customers, or invite them to customer meetings in HQ. Give them the context for the decision.
There will always be people who cannot accept the no, and are convinced that you are on the wrong path. Again, you need to accept that there are just some people who refuse to see the bigger picture, no matter how hard and long you try - if they have political power you may find a backlash brewing - so try to be aware of the environment and keep your management briefed. This has been the source of some of the most difficult moments in my career, so don't feel bad if you are in the situation and need to reach out to your manager for help.
The biggest risk is where saying no to a customer will kill the deal, especially if the deal is large. You really need to ask if this is the type of customer and market that fits where the company is strategically focused. One company I worked for had a solution (set of features) that had not been worked on for many years, but were selling well in one regional market. There were numerous other companies fully focused on this solution, and it wasn't part of our strategic direction. Product Management was constantly asked to present a roadmap to these customers, but our plans had no new features in this direction - there just wasn't the business to justify the engineering investment. Initially we told the customer we regarded the feature set as mature and that the product was fully supported in this mode of operation (i.e. the truth). If the customer needed new capabilities we'd steer them to another company with a better solution - and the customer was happy with our honestly and openness. We were solidifying a long term relationship, leading to better sales in our core markets and solutions.
A few additional tips/recap:
- Listen, and show empathy
- Be consistent but don't be afraid to admit that you were wrong
- Don't abuse your power or their may be a backlash
- Recognize when things may be escalated, and prepare management
The biggest challenge is likely to come from Sales, due to their focus on quarterly results and quotas driving income. The hardest period is between starting the role and delivering your first results, so focus on key relationships and get them as comfortable as possible. Once you have a reputation for delivering, life becomes much easier, and the no's are accepted much more readily.