Ok I got it working. You can use public key authentication but you can't generate them from the switch.
1) Generate RSA keys in linux server using the following commands
ssh-keygen -f sub_rsa_1024 -t rsa -b 1024
2) Now copy-paste the sub_rsa_1024.pub (public file)contents using the following command and associate it with admin user.
Switch# create sshd2 user-key sub_rsa_1024 AAAAB3NzaC1yc2EAAAADAQABAAEXOSisCOOL80aYjF1rpveAyFzCHhMJp7N61a43FY7sZPnxQpkSxjsuJ/gda2D+biiYmd3bpinGtcd+k3mANk4K+LT/wtA4I0wStF5eT5Jg8aN5HPEMdhvHhPJH1IodeQDotqfRRXAup4IgYvk5eT/ndYDRzqKsgwuNKO8kwTUgw==
Switch# configure sshd2 user-key sub_rsa_1024 add user admin
3)Try logging in using the user with the key from the SSH client
ssh -i /root/.ssh/sub_rsa_1024
admin@192.0.2.100
Logs from the switch:
09/30/2015 10:08:07.94 Msg from Master : Did key authentication for user admin (192.0.2.200)
09/30/2015 10:08:07.94 Msg from Master : Login passed for user admin through ssh (192.0.2.200)
09/30/2015 10:08:07.94 Msg from Master : Found valid key for user admin
SW login:
login as: admin
Authenticating with public key "rsa-key-20150930"
ExtremeXOS
Copyright (C) 1996-2015 Extreme Networks. All rights reserved.
This product is protected by one or more US patents listed at
http://www.extremenetworks.com/patents along with their foreign counterparts.
==============================================================================
Press the or '?' key at any time for completions.
Remember to save your configuration changes.
Switch#