Fail to deposit your payroll taxes on time and the IRS can go after anyone in the company it considers to be a responsible person. The IRS recently lost its bid to hold an office manager responsible after a federal trial court ruled that whether she was a responsible person was a matter for a jury to decide.