Measurement of heavy-quark jet photoproduction at HERA