Yes, you can hide your directory from other members. You can do this by going into your Settings, selecting Community Sidebar, and toggling off User Directory.

If you want to show the directory only to certain users, you can toggle on Limit Access? and then choose who or which groups can see the directory.