Can you crawl other sites and use there videos?

[webdev]

I am interested in creating a video site, and was thinking about designing a script which crawls a select amount of sites and then streams the videos of those sites on my new site? Is this legal on both adult/porn type videos? and also other types of videos? Or do some sites allow it and other...