Despite the astounding results we have seen from transformers and language models in academia, deploying these models in industry is still very challenging because: 1) maintaining these models is very time and cost consuming 2) there is not yet enough clarity about when the advantages and challenges of these models outweigh classical ML models. In this talk, we discuss these challenges and share our findings and recommendations from working on real world examples at SPINS.