Season 4 is actually regarded to be one of the better seasons of the show, the beginning of season 3 was awful though. Also they do flashbacks throughout the whole show
By any chance, did you watch it when it first came out? Lost was made to be binge watched in an era of television before that was commonplace. It holds up much better if you can watch an episode every few days, instead of once a week.