Climate is both an enabler and disruptor of human endeavors. Nature balances the population it supports. Technology enables increased rate of energy extraction from environment to further human purposes.
Conducive climate results in bountiful yield and has allowed Rome to rise. Land is cleared and trade networks flourish during the rise of the empire. Clearance of wild lands unlocked microbes into the human civilization.
Microbes evolve to utilize humans and other mammals as vectors of infection. Dense population and connected trading network serve as a multiplier. The worst pandemic is the white pestilence (Black Death). This wiped out more than half of human population.
Trading network collapse as a result of the decimated population. Military rank diminishes both as a result of population decimation. Problem is further compounded by collapse of financial system which makes it difficult to sustain an army.
Grounds are fertile for spread of monotheistic dooms day religion like Christianity. Emperor Justinian converts to Christianity. Classical Greek school of thoughts gets displaced.
Climate change for the worse force nomads with superior military power to migrate westwards into Roman territory.
Roman Empire with decimated population gets further crippled.