Canadian moving to the US on TN with children

Hello,

Well I've posted before regarding our move to Florida on a 3 year TN work visa. My wife has no aquired herself a position working in one of the largest Cardiac Care Hospitals in the area, she has her Florida Nursing License and her CGFNS VS Cert and all required documents for the move. We intend to go to the POE in the next week to get the Visas and start the temp. move south!

Now my question is regarding our children, or I suppose I should say her children, but they are like mine really! My wife has sole-custody of the children agreed to by her ex and awarded by the courts, her ex is aware of our move (notification sent over 6 months ago) and he has signed a document to say he has no problems with it as he saw the children at best once a year anyway (his choice). It is also stated clearly in the Custody Order that he has no issue with us moving either within or outside of Canada at any time and the decision will be my wife's (Obviously if he chooses to oppose it within 30 days he can but the courts have said it would be fairly futile considering his history) Now my concern is that everyone I speak to here tells me the CBP will call him after reading the order and the consent to move letter, they say that we will not be allowed to leave until we have spoken with him!! Is this true? He NEVER answers his phone, when I wrote to him last giving him our date of travel he wrote back to my lawyer saying he didn't care as he was leaving the country himself before then! So if he leaves I'm worried we will now be declined our Visa's as he is not around to 'approve' it with... I just don't see this being the case as we have a Court Order clearly stating we can move, that my wife is Sole Custodial parent and a letter saying he know's we're moving and is ok with it!... Am I being told the truth? If so I am at a bit of a loss...

I would appreciate any help you can offer! This site has been SUCH a big help in the past I thank you all!