The difference between custom instructions that work and custom instructions that the tool seems to ignore comes down to specificity and clarity. Three principles will serve you well.
First: be specific rather than general. “Write well” is not an instruction. “Write in a warm, professional register that reads as an advisor speaking to a valued client, not a brand speaking to a consumer” is an instruction. The more specific you are about what you want, the more consistently the tool delivers it.
Second: use concrete examples of what to avoid. “Do not be too formal” is vague. “Do not use phrases like ‘I hope this finds you well,’ ‘please do not hesitate to contact me,’ or ‘we are delighted to present.’” That is concrete. The tool can comply with specifics far more reliably than with generalities.
Third: test and refine. Write your custom instructions, use the tool for a week, and note every time the output does something you do not want. Each of those moments is a missing instruction or a vague one that needs to be sharpened. Custom instructions are not something you write once and forget. They are a living document that improves as you use it.