The discovery of the Higgs boson by ATLAS and CMS has opened a new frontier of exciting measurements in Particle Physics. The tau lepton, a peculiar cousin of the electron, offers the opportunity to study a new type of force associated with the Higgs boson: the Yukawa interaction. However detecting the tau lepton is experimentally very challenging as it decays almost instantly and predominantly converts into a set of pions and an undetectable neutrino. In this talk, I will discuss the strategy employed by the ATLAS experiment to identify hadronic decay products of tau leptons and the modern machine learning algorithms employed to distinguish them from QCD jets. I will then present the latest ATLAS results on Higgs boson measurements with tau leptons and discuss their long term prospects with the upcoming data-taking planned with the LHC. Finally, I will present the first result from ATLAS on the anomalous magnetic moment of the tau lepton, a quantity measured with astonishing precision for the electron and the muon.