I have been involved in discussions like this. In my experience it often comes down to someone saying, in not so many words, "I know more about how to secure a database than how to secure a file system."
If users should only have access to some of the files in question, or maybe none, then either (a) they shouldn't have shell or file-sharing access to that server at all, or (b) they shouldn't have shell or file-sharing access to the parent of the directories that contain the files in question, or anything below the parent. Preferably, as this is a back-end server, they will have no access at all. Then the only way they should be able to get at the data is via the application which should have its own authorization system and logic. And in that case, at least from the perspective of logical access, it doesn't matter to the user whether the file data is in the file system or database.
Either one can be as secure as you need it to be. All modern file systems have granular and robust access controls. To me the difference between secure and insecure is people, policy, and process, rather than the technology. Either a database or a file system can be secure or insecure, however you measure that, depending on how it is configured. So security shouldn't be the only factor that impacts the architecture decision of where to store this data.
You also need to know and factor in things like how much of this data there will be, how large it will grow, how long it must be retained (is there a retention policy?), how essential it is to the application (i.e. availability SLA), what your replication needs are, what your performance needs are, etc. For example, in a disaster, can you bring up the production database first and take time to bring up a LOB database that might take a lot longer to be restored or go through crash recovery? Or must the LOBs be available whenever the application is? From that perspective, having that data in the file system is advantageous. As long as the file system is not corrupted, you can access it. Whereas if it's in a database, we know there are a number of scenarios that can affect the availability of database data.
If it were me, I'd put those large files in the file system. But if you decide to use BLOBs instead, I'll echo Cringer's comment: put them in their own database.