Staff vetting

I mentioned vetting in my previous post and I'd like to expand on the vetting of staff, contractors and temporary workers.
I have been through some pretty heavy vetting. I'm glad to say I passed but not without a warning or two - a sign of a mis-spent youth. The vetting process was quite in-depth and included background checks, reference interviews and a face-to-face interview.
During my vetting, I was completely honest and advised my references to be just as honest. I have encountered some managers advising their staff to be less so, particularly in certain areas. This was especially apparent when working in the private sector, providing services to HMG. If consultants did not pass their vetting, they were unable to work on ANY projects for HMG therefore the pressure to pass the checks was quite high. Don't get me wrong, I don't think the people that were lying were any higher risk to security than those that didn't, if I did, I would have spoken up but the vetting scheme requires complete honesty in order that you don't leave yourself open for blackmail, those that lie have this vulnerability.
Why do they lie? It is apparent that, depending on the area you are being vetted for, the tolerance levels for certain behaviours is higher, or lower, than others. Some areas would have no tolerance for alcohol abuse where others may take a softer approach. Some areas my be more tolerant of a past instance of employee theft where others may not.
What does this mean? To put it succinctly, there are a number of people working for HMG either directly or via contract, that hold high levels of clearance, that are (slightly) vulnerable to blackmail.
I am acutely aware that those that are likely to disclose information would also lie to protect themselves and to achieve clearance and it is obvious, at least in my experience, that the vetting staff are not trained to spot obvious body language tells that would indicate a less honest answer.
The alternatives are unpaletable. I understand anecdotally that in the US, vetting is accompanied by a lie detector test. I do not believe that would be acceptable in the UK and would certainly be rejected by many. I am also aware that it is possible to be trained to defeat the lie detector just as easily as one can be trained not to provide body language tells. If we were to rely on further reference interviews, the costs would soon increase - vetting is not a cheap process and I'm sure the vetting agencies are under pressure to keep costs down and if one reference will lie for you, I'm sure many others would also.
The bottom line is that many potentially good candidates are put off by the draconian requirements for vetting and some previous behaviour will preclude a candidate from attaining the highest level of clearance where other candidates who are prepared to lie are achieving clearance, leaving them open to blackmail. I elieve the system is broken and there is no easy fix.