In Macbeth, is Macduff irresponsible in leaving behind his family when he flees to Malcolm?

Lady Macduff certainly thinks so.  In Act 4 Scene 2, Lady
Macduff asks Rosse why her husband has fled the region.  Rosse does not want to give
away any information, so he tells her that she must trust Macduff's judgement.  But Lady
Macduff thinks that he is a coward for leaving his family behind unattended and without
any apparent reason.  After Rosse leaves, the murderers enter and slaughter Macduff's
entire family and court.

Of course, Macduff had no way of
predicting that murderers would enter his castle to kill his family; however, he is
aware that Macbeth is on a killing spree and he does suspect that Macbeth is suspicious
of him.  These should have been clues to Macduff that Macbeth may have been plotting to
do him harm.