You have a lot of attack vectors as a customer using a VPS.

Providers can really easily peek on what you do with stuff like OpenVZ virtualization.

So I'd avoid OpenVZ if privacy is necessity (it rocks for other stuff though).

KVM is likely what you will want to use.

Best to get ISO install of OS from legit distro. Pre bundled easy to install options providers have there for 1 click style installs should be avoided. Might find users in there already by default as backdoors (this just happened this week with OpenVZ template for Debian 10 - and from the official source allegedly - meaning many providers were backdoored).

You also want a provider who is accustomed to and appreciative of privacy conscious customers. Most providers want nothing to do with such. They are all about in creepy way often knowing their customers too well (but they never say to you that are profiling and spying on you).

For payment, shield yourself with prepaid cards (Visa, Mastercard, etc.).

Drop your personal details. Create a persona with info that checks out long before signing up. Give your little foot soldier their own Twitter and randomly pile stupidity in there. Create a free privacy email address just for this use.

Location is another thing. 5/19/14 eyes are to be avoided. However, there is legal nexus between any business doing biz anywhere and then also doing biz in one of those countries. Meaning the shop with 5 locations, one of which is private while 4 are in 5/9/14 countries has little to do but comply with 'authorities'.

Basically on provider side you end up dealing with either totally privacy focused companies or one of a handful of long time in the trench companies. Those companies can be good. But know you are moving into a sketchy neighborhood. If you are hosting legit content there, might be algorithmic bias and rank drop in search and other punishments from the corporate net controllers.

As far as naming a company, not endorsing anyone freely or otherwise. I have a few I have used for a decade plus. Emphasis on long term existence and durability of shops vs. cheap hobby hosts that come and go and are useless unless downloading piracy and bulk data and onto the next one quickly.

Storage - that's big deal with VPS. Make sure your OS is boot time encrypted so it's not simple to spin it up. LUKS is solution for this and baked into many OS installs today as option.

BUT! That's solution for boot. You need a second volume otherwise encrypted for your critical data. Isolation is a thing. Different crypto, different keys.

Providers that offer mapable storage - block or other forms of storage are good for your use. Confuses things a bit more and can bring those up and down often in short notice. So you decouple to some extent your vital stuff from the OS itself.

Lots to consider.

You cannot guarantee this with a VPS, no matter what the provider says