Should I keep receipts if the charge appears on a bank statement?

Yes. A bank statement shows a payment, but often does not show what was purchased or its business purpose. Keep the underlying receipt or invoice.