Consensus in the field seems to be that PyTorch is superior for other reasons than performance - mostly due to more intuitive workflow and dynamic graphs. That said I’ve noticed the same thing you have here, in head to head comparisons tensorflow tends to be a hair better when all is said and done. But workflow matters a lot to many.

If you can get data into XGBoost though, I suspect it would outperform both. Neural networks are great, but I’ve very rarely seen one beat gradient boosting when data permits the use of either.