
Okay, figured it out. The device enumeration is a bit odd. I tested each device number starting from 0 and got the following.
BASS_RecordStart() errors
when _inputDevice =
0 -> BASS_ERROR_FORMAT
1 -> BASS_ERROR_UNKNOWN
2 -> BASS_ERROR_DRIVER
3 -> BASS_ERROR_DRIVER
[89722.005157] OMAP ABE Voice: asoc: OMAP ABE Voice no valid capture route from source to sink
4 -> BASS_ERROR_DRIVER
[90186.655029] OMAP ABE MODEM: asoc: OMAP ABE MODEM no valid capture route from source to sink
5 -> work but soon crash.
8 -> BASS_ERROR_DRIVER
9 -> BASS_ERROR_DRIVER
From the OMAP errors I was able to determine the device number is enumerated according to the capturable devices on the system starting from 1 (not 0). Since OMAP ABE Voice is Card 0 Device 2 and OMAP ABE MODEM is Card 0 Device 5 on my Pandaboard.

Only problem now is my kernel hits a NULL pointer dereference while recording:
[ 72.340393] Unable to handle kernel NULL pointer dereference at virtual address 00000008
[ 72.351654] pgd = ed6ac000
[ 72.354553] [00000008] *pgd=ad6fe831, *pte=00000000, *ppte=00000000
[ 72.361358] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
[ 72.367034] Modules linked in: ftdi_sio bnep btwilink bluetooth omapdce(C) omap_rpmsg_resmgr rpmsg_resmgr virtio_rpmsg_bus omap_remoteproc remoteproc wl12xx
[ 72.381866] CPU: 1 Tainted: G C (3.4.12 #1)
[ 72.387573] PC is at snd_pcm_info+0xd4/0xe4
[ 72.391998] LR is at 0x30232065
[ 72.395294] pc : [<c03d3d60>] lr : [<30232065>] psr: a0000113
[ 72.395324] sp : ed683df8 ip : ed637ed0 fp : ed683e14
[ 72.407440] r10: 00000009 r9 : ef0cc400 r8 : c08040b0
[ 72.412933] r7 : ef3c88a8 r6 : ef3c8800 r5 : ef3c8a00 r4 : ed637e00
[ 72.419830] r3 : 00000000 r2 : ed637e00 r1 : 00000001 r0 : ef3c8a00
[ 72.426727] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 72.434295] Control: 10c5387d Table: ad6ac04a DAC: 00000015
[ 72.440338]
...
[ 73.484588] Backtrace:
[ 73.487182] [<c03d3c8c>] (snd_pcm_info+0x0/0xe4) from [<c03d3dc4>] (snd_pcm_info_user+0x54/0xbc)
[ 73.496459] r7:00000000 r6:ef3c8a00 r5:b49fe670 r4:ed637e00
[ 73.502471] [<c03d3d70>] (snd_pcm_info_user+0x0/0xbc) from [<c03cfa68>] (snd_pcm_control_ioctl+0x248/0x26c)
[ 73.512756] r6:ef0cc400 r5:00000000 r4:b49fe670 r3:00000000
[ 73.518768] [<c03cf820>] (snd_pcm_control_ioctl+0x0/0x26c) from [<c03cb698>] (snd_ctl_ioctl+0xa08/0xbbc)
[ 73.528778] [<c03cac90>] (snd_ctl_ioctl+0x0/0xbbc) from [<c010ef18>] (vfs_ioctl+0x38/0x4c)
[ 73.537506] [<c010eee0>] (vfs_ioctl+0x0/0x4c) from [<c010fac4>] (do_vfs_ioctl+0x544/0x598)
[ 73.546234] [<c010f580>] (do_vfs_ioctl+0x0/0x598) from [<c010fb5c>] (sys_ioctl+0x44/0x70)
[ 73.554901] [<c010fb18>] (sys_ioctl+0x0/0x70) from [<c000e500>] (ret_fast_syscall+0x0/0x30)
[ 73.563690] r7:00000036 r6:b49fe7c0 r5:b51280b0 r4:00000000
[ 73.569702] Code: e1a00005 e595309c e3a01001 e1a02004 (e5933008)
[ 73.577087] ---[ end trace 46efbe5582039872 ]---
I'll tinker with the code and hope the error is not from the lib.
Any ideas please. I posted the code earlier, but changed the _inputDevice = 5;
Thanks.