Toronto's fire chief says one person died and five others were taken to hospital after a fire at a downtown community housing building on Thursday evening.

Matthew Pegg told a late-night news conference that the fire broke out in a unit on the fourth floor of the five-storey building on George Street, near Dundas Street and he believes the victim was found in the unit where the fire started.

He said the top two floors were evacuated and some people were being sheltered on the lower floors.

However, Toronto police later announced via Twitter that the entire building was to be evacuated and residents taken to temporary lodging until the building is inspected.

"Until we can get our investigation teams inside, we won't have any idea as to when we can get people back in." Pegg said. 

Firefighters led some people out of the building, but they also had to use ladders to rescue some stranded residents.

"Our crews were able to bring some people out through the inside of the building ... other people brought down on ladders, so a very complex, very demanding rescue," said Pegg.

He also said there was no immediate information on the age or gender of the victim, nor on the conditions of the people taken to hospital.

As for the cause of the fire, Pegg said it was "far too early to speculate as to origin and cause."

Pegg said the Ontario Fire Marshal's office had been notified and will assist in the investigation.

The Canadian Press