If you intend to have sendmail support nisplus maps, you need to define NISPLUS in your Build m4 file (the use of nisplus aliases and other maps is determined by the /etc/nsswitch.conf file):

APPENDDEF(`confMAPDEF', `-DNISPLUS')

If NISPLUS is defined, the AliasFile option can be used to override the setting of the /etc/nsswitch.conf file:

O AliasFile=nisplus:mail.aliases  V8.7 and above

Here, nisplus aliases will be used even if the /etc/nsswitch.conf file excludes them.

See Section 24.9 for details about the AliasFile option. Note that NISPLUS is new beginning with V8.7 and not supported under earlier versions of sendmail.

With NISPLUS defined, support is also included to declare and use nisplus-type maps (Section 23.2.2) with the K configuration command.

If you are running a precompiled sendmail binary, you can use the -d0.1 debugging command-line switch (-d0.1) to determine if NISPLUS support is included (if it appears in the list, support is included).