A security firm claims to have found a flaw in the sandboxing functionality of Mac OS X.

Mac App Store sandboxing is coming in March 2012, as Apple announced last week that all apps submitted to the Mac App Store will have to implement the system in order to gain approval.

'Sandboxing' restricts the system resources available to an app, meaning that the app can't execute certain commands that would control other software or parts of the operating system.

However, CoreLabs Research thinks that it has found a flaw in the system could allow rogue apps to excecute commands without the knowledge of the user. The vulnerabilities, it said, exist in Mac OS X Leopard, Snow Leopard and Lion.

"Several of the default pre-defined sandbox profiles don't properly limit all the available mechanisms and therefore allow exercising part of the restricted functionality. Namely, sending Apple events is possible within the no-network sandbox (kSBXProfileNoNetwork)," CoreLabs' assessment of the vulnerability reads.

"A compromised application hypothetically restricted by the use of the no-network profile may have access to network resources through the use of Apple events to invoke the execution of other applications not directly restricted by the sandbox."

Paul Ducklin of rival security firm Sophos explained CoreLabs' claims but said he was sitting on the fence as to whether or not he believed the flaw to be problematic.

"The criticism from Core Labs is that, whilst sandbox restrictions apply recursively to processes directly spawned by a sandboxed application, they don't apply to processes spawned indirectly. So, for example, you can use Apple Script to tell OS X to start some other arbitrary program (or a second copy of your own) which won't inherit your sandbox settings," Ducklin said.

However, one of the commenters on Ducklin's blog expressed doubts that CoreLabs' assessment of the issue was correct.

"The "no network" policy simply enforces that a process and its children will not be allowed to create socket connections. It does not restrict access to AppleEvents, filesystems or anything else. The problem is not that the "no network" policy is doing something wrong, the problem is that "no network" is not enough," wrote the visitor going by the name of 'snare'.

"All they're doing in this PoC is telling some other process, using a system IPC method, to do something on their behalf. Any application can send AppleEvents directly using the APIs without resorting to calling osascript(1). Shellcode could quite feasibly send AppleEvents directly to launchd or some other process even if the sandbox policy restricted its ability to write to the filesystem or execute osascript."

CoreLabs has reported the issue to Apple. According to CoreLabs, Apple said it "does not see any actual security implications" though Cupertino has not made any official public statement about CoreLabs finding.